Today’s Bogus Stats Award in the amateur category goes to Megan McArdle, who claims, in an Atlantic article called How Real are the Defects in Toyota’s Cars? to have discovered that the “overwhelming majority” of the sudden acceleration incidents involved people “over 55.”
First of all, this isn’t even clear from the graph she provides, which has a 50-60 category, but nothing that cuts off at 55 specifically. Here it is (without permission, but also available at the original site):
And second of all, I count 20 – not 24, as she claims elsewhere – out of 35, which is a clear majority, sure, but not really an “overwhelming majority.” It’s 57%. Now, granted, this total includes 5 unknowns, and if we extrapolate from the knowns (which I don’t recommend, but just for shits and giggles), in which case older people are 20 in 30, or 67%, then we would expect about 3 of those unknowns to be above 50 – i.e. 23/35, or 66%. And 66% is nothing if not an “overwhelming majority.” Not to mention, people over the age of 50 are, according to the 2000 census, only about 27.3% of the total population. Even if we take that with the grain of salt that fully 10% of the population is below driving age, the over-50 category is still less than half of the population but is accounting for almost 70% of these accidents – WAY more than their fair share. So McArdle is on to something, right? There’s a high correlation between age of driver and these incidents?
Yeah, not so much. This is a classic example of what I like to call the “baseline fallacy,” and it works like this: you take a completely legitimate sample (I have no reason to believe that McArdle is cooking her numbers – no doubt the demographics of these incidents work out just as she claims) and conclude things about the relations between arbitrary subsets of that sample BUT – and this is the key step – you neglect to take into acount how unlikely membership in the sample is to begin with. No matter how you cook these numbers, we’re still talking about 35 affected drivers out of millions of Toyota drivers. So yeah, when you say 23/35 affected drivers were over 50, aka 66%, that looks significant. But when you put these numbers in perspective? I can’t find any exact numbers on how many Toyotas are on the road vs. other kinds of cars, but in 2006 about a million Japanese cars were sold in the US, and there are close to 300 million vehciles registered in the US, just under half of which (about 136 million) are passenger cars. So, let’s just throw up a number of something like 100,000 Toyota cars per year for the last 15 years and just arbitrarily decide that 1.5 million of the 136 million autos on the road are Toyotas. That srikes me as a GROSS underestimation, so I think I’m being fair when I say that if we’re really talking about 23/1.5million drivers who are elderly and 12/1.5million who are not, then your chances of having this happen to you if you’re old are 0.000015% and 0.000008% if not. The chances are so vanishingly small for both groups – because 35/1.5million is only 0.000023% of all Toyota drivers, even by my bogus and overly generous estimation – that I would really hesitate to make anything out of this. I would feel more comfortable saying that when you pull 35 marbles out of a 1.5million-marble jar, it sometimes happens that 23 of them are red and not blue is all.
McArdle’s point seems to be that because older people are slower to react (and she also throws in some numbers that indicate that most of the incidents happen when stoping and starting back up), that there’s no real problem here. It’s just the molehill of old people getting confused while driving being blown up into a mountain. I agree that this is basically a nonissue (Toyota should, of course, proceed with the recall, but the lawsuits are all frivolous, and I really very seriously doubt that Toyota has been just sitting on the knowledge that there might just maybe be a sudden acceleration problem without doing anything about it) – but NOT because there’s one way of looking at the data that provides some exremely slight statistical evidence that mostly old people have this problem! I say “one way of looking at the data” because to do this properly she would have to show that old people weren’t disproportionately likely to be Toyota owners – which they might be – that old people don’t clock more road hours – which they might well, and so on. There are a lot of potentially significant variables unaccounted for here. No, the reason this is a nonissue is just because it doesn’t seem to happen very often. Her basic point is solid:
At any rate, when you look at these incidents all together, it’s pretty clear why Toyota didn’t investigate this “overwhelming evidence” of a problem: they look a lot like typical cases of driver error.
Right. When no more than 35 of your millions of customers call to complain about sudden acceleration, and the damage in most cases is minimal because it happened in parking lots, you do rather tend to assume that the person in question probably just stepped on the gas by accident. ESPECIALLY when making the opposite assumption with little to no evidence would cost you billions precisely when the economy is troubled and your competitors are getting free money from the government! No, at best Toyota’s guilty of political stupidity; they’re certainly not guilty of callousness – at least, I don’t see any evidence for it. So let’s cut the crap about spurious correlations, shall we?
I like to think that McArdle knows what’s wrong with her article, and she wrote it partly to have something to publish, and partly because the “safety über alles” crowd won’t be convinced by the actual facts (they truck in this kind of statistical fallacy for a living, after all). But even so, she’s doing her readers a disservice by not telling them what she’s up to. Now, instead of just the “safety über alles” crowd playing useful idiots for the Big Three, even Toyota’s defenders are getting their stats wrong. I don’t know when political discourse first got moronic – probably it was ever thus – but I do know this isn’t helping.