Bad polls: Headed for YOUR pages?
It's early in the election season, so let's see if we can put a stop, once and for all, to any references to the so-called "RCP Average," a purported average of surveys showing how the major parties' major candidates are faring. So here goes:
** Do not use the RCP Average. It is a meaningless number. **
The thing about numbers is that we rely on them to represent the same sorts of relationship from case to case and from day to day. "Twenty" is always going to be twice the size of 10. "34.2" is going to be the same distance from "21.8" today that it was yesterday. Forever and ever Amen.
Not so with the RCP Average. It's a number, all right, but there's no guarantee it represents what it claims to. Under its rules, "19.5" might mean exactly what we always thought. Or it might not. Let's look at why, using today's campaign coverage from the McClatchy Washburo:
Averaging four Iowa polls taken since late September, Clinton leads with 26.8 percent, Obama is second with 23.8 percent and Edwards is third with 21.5 percent, according to RealClearPolitics.com.
And if you don't believe us, you can head right over to Real Clear Politics and see for yourself! Right? Well, not so fast, for two reasons. First, RCP is apparently summing each candidate's percentage in the four polls, dividing the results by 4 and presenting its readers with mean scores for each candidate. That's the wrong tool to use, and it risks making the writer look -- take your pick -- either careless or biased.
One reason the "mean" is such a widely used measure is that it's equally sensitive to every point in the distribution. That's fine if you're averaging your score on 10 weekly quizzes, each worth 5 percent of your grade. It's really, really bad if you throw in your grade on the midterm (20 percent) and the final (30 percent) and just divide the whole thing by 12. That's what RCP is doing (and it's also one reason why reporting the fine points of any survey you run is essential to your credibility).
What we need here is a weighted mean -- not one that's equally sensitive to every survey, but one that gives more weight to surveys with bigger samples and less to ones with smaller samples. Instead of summing the percentages themselves, multiply each percentage by the sample N (so 38% among a sample of 600 would be 38x600) and add those results. For the denominator, instead of adding up the number of polls, add the N's of the polls (so for four polls of 600 each, you'd use 2,400 rather than 4).
Sometimes the result is almost exactly what RCP gets. For six polls in New Hampshire, Clinton's RCP average is 39.3, compared with a weighted mean of 39. Not bad. And Obama comes out at 19.5 by both measures. But agreement by chance is a matter of luck, and luck isn't an appropriate survey technique. The results of the RCP average of national surveys show why.
RCP gives averages of 45 for Clinton and 23.8 for Obama. We can't use those, since RCP doesn't bother to report a sample size for one of the five polls (eat your Wheaties, kids, and always report those sample sizes), so we'll recalculate without the missing data: 45.5 for Clinton, 24.25 for Obama. Once again, Obama's weighted mean is right on target, but not Clinton's: She's at 49, or 3.5 points above the corrected sample and 4 points (about 9 percent) above what RCP claims.
That's half the problem: It's possible to compare polls of different sizes, but it takes more work than just adding the percentages and dividing by 4. Part two is uncorrectable: Comparing unlike samples. If you look over the past four weeks' worth of national polls for the Democrats, you'll see three groups: Likely voters, registered voters and adults. They overlap a lot, but they aren't the same set. Skill at shooting free throws tends to correlate highly with field goal percentage, but you can't add them, divide by two and get a meaningful concept called "basketball shooting average."
There's a lot more to worry about in the story at hand: Why the Des Moines Register poll is worth being in the lede for a second day in a row. Whether we're giving altogether too much weight to surveys at a point at which they really don't say anything very relevant. Why Obama is "bragging" while Clinton is "declaring." Whether it's appropriate for the media to join in anointing the alleged front-runners this early, or whether it needs to be reinflating the Public Domed Stadium of Debate. But let's start by throwing out the stuff we know is wrong.
** Do not use the RCP Average. It is a meaningless number. **
The thing about numbers is that we rely on them to represent the same sorts of relationship from case to case and from day to day. "Twenty" is always going to be twice the size of 10. "34.2" is going to be the same distance from "21.8" today that it was yesterday. Forever and ever Amen.
Not so with the RCP Average. It's a number, all right, but there's no guarantee it represents what it claims to. Under its rules, "19.5" might mean exactly what we always thought. Or it might not. Let's look at why, using today's campaign coverage from the McClatchy Washburo:
Averaging four Iowa polls taken since late September, Clinton leads with 26.8 percent, Obama is second with 23.8 percent and Edwards is third with 21.5 percent, according to RealClearPolitics.com.
And if you don't believe us, you can head right over to Real Clear Politics and see for yourself! Right? Well, not so fast, for two reasons. First, RCP is apparently summing each candidate's percentage in the four polls, dividing the results by 4 and presenting its readers with mean scores for each candidate. That's the wrong tool to use, and it risks making the writer look -- take your pick -- either careless or biased.
One reason the "mean" is such a widely used measure is that it's equally sensitive to every point in the distribution. That's fine if you're averaging your score on 10 weekly quizzes, each worth 5 percent of your grade. It's really, really bad if you throw in your grade on the midterm (20 percent) and the final (30 percent) and just divide the whole thing by 12. That's what RCP is doing (and it's also one reason why reporting the fine points of any survey you run is essential to your credibility).
What we need here is a weighted mean -- not one that's equally sensitive to every survey, but one that gives more weight to surveys with bigger samples and less to ones with smaller samples. Instead of summing the percentages themselves, multiply each percentage by the sample N (so 38% among a sample of 600 would be 38x600) and add those results. For the denominator, instead of adding up the number of polls, add the N's of the polls (so for four polls of 600 each, you'd use 2,400 rather than 4).
Sometimes the result is almost exactly what RCP gets. For six polls in New Hampshire, Clinton's RCP average is 39.3, compared with a weighted mean of 39. Not bad. And Obama comes out at 19.5 by both measures. But agreement by chance is a matter of luck, and luck isn't an appropriate survey technique. The results of the RCP average of national surveys show why.
RCP gives averages of 45 for Clinton and 23.8 for Obama. We can't use those, since RCP doesn't bother to report a sample size for one of the five polls (eat your Wheaties, kids, and always report those sample sizes), so we'll recalculate without the missing data: 45.5 for Clinton, 24.25 for Obama. Once again, Obama's weighted mean is right on target, but not Clinton's: She's at 49, or 3.5 points above the corrected sample and 4 points (about 9 percent) above what RCP claims.
That's half the problem: It's possible to compare polls of different sizes, but it takes more work than just adding the percentages and dividing by 4. Part two is uncorrectable: Comparing unlike samples. If you look over the past four weeks' worth of national polls for the Democrats, you'll see three groups: Likely voters, registered voters and adults. They overlap a lot, but they aren't the same set. Skill at shooting free throws tends to correlate highly with field goal percentage, but you can't add them, divide by two and get a meaningful concept called "basketball shooting average."
There's a lot more to worry about in the story at hand: Why the Des Moines Register poll is worth being in the lede for a second day in a row. Whether we're giving altogether too much weight to surveys at a point at which they really don't say anything very relevant. Why Obama is "bragging" while Clinton is "declaring." Whether it's appropriate for the media to join in anointing the alleged front-runners this early, or whether it needs to be reinflating the Public Domed Stadium of Debate. But let's start by throwing out the stuff we know is wrong.
Labels: polls
posted by fev at 7:30 PM
2 Comments:
RCP is apparently summing each candidate's percentage in the four polls, dividing the results by 4 and presenting its readers with mean scores for each candidate.
I thought you were joking.
That's the only way I can get all the arithmetic to come out consistently the way RCP presents it. Tell me if I'm missing something.
Post a Comment
<< Home