Polling sins: Heads are surging!
I think this is another of those gems for which we can thank Darrell Huff, author of "How to Lie With Statistics": The story about the guy who didn't believe a story he read in the paper, so he went out and bought a dozen more copies to make sure. That's the only way I can explain this bizarre interpretation of an otherwise nondescript poll result, which seems to have cropped up across the nation:
Here's a WashPost blog:
Rick Santorum (4-1): A Des Moines Register poll released Saturday night made plain that the former senator from Pennsylvania is the momentum candidate. Although he took 15 percent overall in the four-day survey, he was at 21 percent in the final two days — a sign that he is peaking in the waning moments.
And the Fair 'n' Balanced Network:
Santorum, who just two weeks ago was polling in single digits, came up third in the Des Moines Register poll. For the final two days of polling, he placed second behind Romney.
The National Review's blog:
In the closely followed Des Moines Register poll just released at 8pm, Romney is 24, Paul 22, Santorum 15, Newt Gingrich 12, Rick Perry 11, and Michele Bachmann 7. That’s the four-day result. If just the last two days are taken into account, Romney is still at 24, Santorum jumps to 21, and Paul is down to 18. That means–in the word of the hour–Santorum has momentum.
The Des Moines Register, condensed in the local fishwrap:
The poll, conducted Tuesday through Friday, shows support for Romney, a Michigan native, at 24%; Paul at 22%, and at 15% for the surging Santorum of Pennsylvania.
But the four-day results don't reflect just how quickly momentum is shifting in a race that has remained highly fluid for months. If the final two days of polling are considered separately, Santorum rises to second place, with 21%, pushing Paul to third, at 18%. Romney remains the same, at 24%.
"Momentum's name is Rick Santorum," said the Register's pollster, J. Ann Selzer.
And Nate Silver's blog at the Times:
However, as The Des Moines Register’s Jennifer Jacobs noted, there is a “twist” in the survey. Over the final two days of polling on Thursday and Friday, Mr. Santorum got 21 percent of the vote, much better than he polled on Tuesday or Wednesday. Mr. Paul, meanwhile, dropped to 18 percent in the final two days of the survey, while Mr. Romney’s standing remained intact at 24 percent.
Because voter preferences in a primary or caucus can be fickle and can change on a near-daily basis, there is a premium in accounting for the most recent information.
Once again, this isn't a partisan development. It pretty much spans the spectrum of mainstream US journalism, from the elite center and center-right "national" papers to Mr. Murdoch's Republican house organ. But it does reflect ideology: here, the journalistic belief that non-evidence-based results are meaningful, as long as enough people say them in the same way. Hence, the obsession with talking about who has "momentum," or who's "surging," without regard to whether "momentum" has been measured or whether the results differ from the previous measurement at better than chance levels.
Here's the problem. More recent information is often at a premium (though it's an ongoing fallacy of journalism to assume that new information is better because it's newer), but the second half of a random sample is not "most recent information." You can illustrate this for yourself by tossing coins.
If heads and tails are the exclusive and exhaustive outcomes of a coin toss,* the safest bet on a series of coin tosses is going to be an even split. That's a better bet the more times you toss the coin, but that doesn't mean later tosses are more accurate than earlier tosses, or that any subset drawn from the second half of your pre-agreed set of tosses indicates an underlying change in how headses and tailses are distributed. You could quite easily get 2 heads and 8 tails in your first 10, then 8 heads and 2 tails on your next,** without either set having any impact on the next toss, let alone the next 80 if you're aiming for 100.
Voter preferences aren't tosses of a coin. They do change, and those changes can be reflected in fairly small samples, and preferences can become more distinct closer to the election. But part of the job in journalism is taking into account the other plausible explanations for what you see. It's not impossible that the Iowa poll detected a significant change in Santorum's support from the first two days to the last two days (which also partly depends on how strict you are about confidence levels). It's equally possible that the two halves of the sample showed the same thing that our 2-8 and 8-2 sets of coin tosses showed: 50-50 is the outcome you'd expect, and the second result doesn't mean heads are "surging."
What you shouldn't do (editors, this means "what you shouldn't let reporters do") is decide which of those choices you report based on what everybody else is doing. A story doesn't become more true when you buy more copies of the paper, or see it repeated more times on CNN, or hear it chewed over on more talk shows. An outcome doesn't become more likely because more people are saying it, or because a particular poll has a good reputation. A good report on the results given above would have either ignored the Santorum split entirely or made clear that it could easily be an accident of sampling.
Here's the HEADSUP-L polling motto, to be recited as often as needed over the next 10 months:
1) What makes polls sexy usually isn't valuable, and what makes them valuable usually isn't sexy
2) Your poll is not an exception
Lather, rinse, repeat.
* We're going to assume the coin is fair and balanced, not Fair 'n' Balanced.
** Funny, that's exactly what we just got. You can test for significant differences in a series of 10-toss sets with the chi-square engine here.
Here's a WashPost blog:
Rick Santorum (4-1): A Des Moines Register poll released Saturday night made plain that the former senator from Pennsylvania is the momentum candidate. Although he took 15 percent overall in the four-day survey, he was at 21 percent in the final two days — a sign that he is peaking in the waning moments.
And the Fair 'n' Balanced Network:
Santorum, who just two weeks ago was polling in single digits, came up third in the Des Moines Register poll. For the final two days of polling, he placed second behind Romney.
The National Review's blog:
In the closely followed Des Moines Register poll just released at 8pm, Romney is 24, Paul 22, Santorum 15, Newt Gingrich 12, Rick Perry 11, and Michele Bachmann 7. That’s the four-day result. If just the last two days are taken into account, Romney is still at 24, Santorum jumps to 21, and Paul is down to 18. That means–in the word of the hour–Santorum has momentum.
The Des Moines Register, condensed in the local fishwrap:
The poll, conducted Tuesday through Friday, shows support for Romney, a Michigan native, at 24%; Paul at 22%, and at 15% for the surging Santorum of Pennsylvania.
But the four-day results don't reflect just how quickly momentum is shifting in a race that has remained highly fluid for months. If the final two days of polling are considered separately, Santorum rises to second place, with 21%, pushing Paul to third, at 18%. Romney remains the same, at 24%.
"Momentum's name is Rick Santorum," said the Register's pollster, J. Ann Selzer.
And Nate Silver's blog at the Times:
However, as The Des Moines Register’s Jennifer Jacobs noted, there is a “twist” in the survey. Over the final two days of polling on Thursday and Friday, Mr. Santorum got 21 percent of the vote, much better than he polled on Tuesday or Wednesday. Mr. Paul, meanwhile, dropped to 18 percent in the final two days of the survey, while Mr. Romney’s standing remained intact at 24 percent.
Because voter preferences in a primary or caucus can be fickle and can change on a near-daily basis, there is a premium in accounting for the most recent information.
Once again, this isn't a partisan development. It pretty much spans the spectrum of mainstream US journalism, from the elite center and center-right "national" papers to Mr. Murdoch's Republican house organ. But it does reflect ideology: here, the journalistic belief that non-evidence-based results are meaningful, as long as enough people say them in the same way. Hence, the obsession with talking about who has "momentum," or who's "surging," without regard to whether "momentum" has been measured or whether the results differ from the previous measurement at better than chance levels.
Here's the problem. More recent information is often at a premium (though it's an ongoing fallacy of journalism to assume that new information is better because it's newer), but the second half of a random sample is not "most recent information." You can illustrate this for yourself by tossing coins.
If heads and tails are the exclusive and exhaustive outcomes of a coin toss,* the safest bet on a series of coin tosses is going to be an even split. That's a better bet the more times you toss the coin, but that doesn't mean later tosses are more accurate than earlier tosses, or that any subset drawn from the second half of your pre-agreed set of tosses indicates an underlying change in how headses and tailses are distributed. You could quite easily get 2 heads and 8 tails in your first 10, then 8 heads and 2 tails on your next,** without either set having any impact on the next toss, let alone the next 80 if you're aiming for 100.
Voter preferences aren't tosses of a coin. They do change, and those changes can be reflected in fairly small samples, and preferences can become more distinct closer to the election. But part of the job in journalism is taking into account the other plausible explanations for what you see. It's not impossible that the Iowa poll detected a significant change in Santorum's support from the first two days to the last two days (which also partly depends on how strict you are about confidence levels). It's equally possible that the two halves of the sample showed the same thing that our 2-8 and 8-2 sets of coin tosses showed: 50-50 is the outcome you'd expect, and the second result doesn't mean heads are "surging."
What you shouldn't do (editors, this means "what you shouldn't let reporters do") is decide which of those choices you report based on what everybody else is doing. A story doesn't become more true when you buy more copies of the paper, or see it repeated more times on CNN, or hear it chewed over on more talk shows. An outcome doesn't become more likely because more people are saying it, or because a particular poll has a good reputation. A good report on the results given above would have either ignored the Santorum split entirely or made clear that it could easily be an accident of sampling.
Here's the HEADSUP-L polling motto, to be recited as often as needed over the next 10 months:
1) What makes polls sexy usually isn't valuable, and what makes them valuable usually isn't sexy
2) Your poll is not an exception
Lather, rinse, repeat.
* We're going to assume the coin is fair and balanced, not Fair 'n' Balanced.
** Funny, that's exactly what we just got. You can test for significant differences in a series of 10-toss sets with the chi-square engine here.
posted by fev at 10:59 AM
1 Comments:
Getting to this a bit late, sorry ...
I'm pretty sure that the folks you quote are right and you are wrong on this one. Your argument that a random sample must only be taken as a whole assumes that the sample will not change over the course of the survey. But especially in the days before a hotly contested election, people change their minds from one day to the next. The data bear this out.
Post a Comment
<< Home