Poll badness: Smoke, mirrors, magic
The polling cluelessness simply isn't going to go away, is it? Let's look at a couple of bad examples of journalism's attitude toward survey research, in hopes that one or two newsrooms at a time will catch on.
The nice thing about polling is that it isn't magic. It relies on a finite set of rules, so it's accessible to anybody. And if you attend to the rules (this is the fun part), you'll be right, no matter what the famous and prominent say. Here's a quick summary of the rules. Then we'll see each one broken in a really annoying way:
1) Some methods of sampling public opinion allow you to make reliable generalizations from a sample to the population it represents. These are called "probability" samples. Generalizing from any other sample is the moral equivalent of reading the horoscopes to your colleagues at lunch: If you're lucky, they'll just think you're harmlessly vacant, not dishonest or irresponsible or anything.
2) Generalizing from samples is about likelihood, not proof. (Want proof? Go to seminary.) A competently reported survey must explain how likely it is that the results could have come about by chance (this is the "confidence level") and the width of the band in which your sample is likely to represent the population (the "confidence interval," better known as the "margin of sampling error"). The "margin of error" is not some magical point at which "statistical ties" become untied; anyone who reports it as such is blowing smoke.
3) Survey research, properly done, is a pretty good way of finding out what people think. It's less effective at predicting what people might do, and that utility can fall off in a hurry based on a number of factors. Never get the two mixed up.
Those are some basic guidelines for hanging around in the empirical world, though of course they don't explain everything. The "scientific method" isn't a guide to running your whole life. It's not a good way to figure out what to have for dinner, or whether you like a movie or a piece of music, or whether you've found your True Love. But if you want to talk about a scientific process like survey research, it's the only one you can use. Sorry, no exceptions.
First up, the McClatchy News Service, illustrating Point 1:
Youths favor Obama, Giuliani
Interesting. And how do we support such a claim?
... Those are the headlines from a national survey of 2,526 likely voters ages 18-24 released Wednesday by Harvard University's Institute of Politics.
Hmm. Big sample for a survey. And it's from Harvard!!! If you'll provide the confidence level, we'll be happy to calculate a margin of sampling error, but ... what's that in the back there?
The survey, taken Oct. 28-Nov. 9 by Harris Interactive for Harvard, was conducted online from a sample of young people who agreed in advance to participate. Because of the method, no formal margin of error was attached to the findings.
That's interesting. "Formal" is not a property of confidence intervals (there's no "informal" way of calculating sampling error to contrast to the "formal" one McClatchy is invoking). And there's only one flavor of sampling that doesn't allow this calculation. Wonder if ... gee, do you suppose it's the same methodology Harris ran for Harvard a couple months ago?
This online survey is not based on a probability sample and therefore no theoretical sampling error can be calculated.
You can stop after "not based on a probability sample," because that means we have no idea whether "youths favor" any of the candidates presented. Doesn't matter how big the sample, and doesn't matter how many times you say "Harvard!!!" That's an argument to (false) authority. Arguments to authority are a good way of finding out whether you can stay out after midnight, but they don't affect the rules of probability. Astrology is still astrology if they do it at Harvard!!!
Next up, Howard Kurtz, representing both the mining-town paper in the town where the mine is government (right, that's the Washington Post) and the fallacy of arguing to tradition. Here's the online Q&A:
Gainesville, Va.: Howard, I think it is time to send political reporters and commentators to a remedial course in statistics. You wrote that Obama could as easily be tied with Hillary because his lead of 28 percent to 25 percent is within the margin of error. What the confidence interval means is that if the confidence interval is smaller than the difference between two means we can be 95 percent (not 100 percent) sure that the opinions held in the universe will reflect the higher response to be greater than the lower. If the difference of the two poll results are within the confidence interval, it may mean we can only be sure that one is greater than the other with 80 percent or 75 percent confidence. It doesn't mean they are tied. It really would be helpful if you and your colleagues boned up on this question before you made statements that aren't true.
Howard Kurtz: I stand by what I said. Obama's lead is within the margin of error. In fact, the poll's margin of error was over 4 percent, which makes the 3-point lead even more of what I would regard as a statistical tie.
This isn't another case in which we have a difference of opinion and need to report both sides. Kurtz is wrong, and the reader is right (though "remedial course in statistics" is excessively snarky, and the reader overlooks the importance of applying the confidence interval to both points in the distribution). Tradition -- "that's the way we do it around here" -- is a fine guide to putting up your Xpesmas goodies, but it's not a part of the scientific method.
Put simply, "within the margin of error" and "statistical tie" are meaningless terms, no matter what Howard Kurtz says (and how any tie could be more of a tie than any other tie is a question for the ages). Let's invent a real-life example: We'll survey 800 registered voters in that hotly contested Crook vs. Liar race and see if we can find out who's ahead. We're going to decide in advance how confident we want to be, and based on that, we'll know how likely it is that what we see in the sample represents a real difference in the population. Let's say our result is Crook 51, Liar 47. Who's ahead, if anyone? Let's apply a few margins of sampling error and see.
Start with 3.5 percentage points. The difference is "outside the margin," so called; what does that mean in real life? Discounting the predetermined likelihood of having gotten our results by chance, rather than the degree to which they represent the population, it means Crook is somewhere between 54.5 percent and 47.5 percent, to Liar's range of 43.5 to 50.5. So we could have a perfectly accurate poll that represents a population in which Liar is ahead of Crook, despite what our results say.
All right, let's say the margin of error is 4.5 points -- meaning Crook's lead is "within the margin of error," as Kurtz (and many others) put it. In real life, again, we're X percent confident that Crook's support is between 55.5 and 46.5 percent, to Liar's range of 42.5 to 51.5. Could Liar be ahead in the whole population? Sure. It's less likely than a Crook lead, but evidently it's perfectly possible.
Well then, let's make the margin of error 1.8 points. Aha! In all nonchance cases, Crook's support is above 49 percent and Liar's is below 49 percent. Except for that agreed-in-advance level that we've set for accidents, we got us a lead.
What's the relationship among those three cases? They're all based on the same results. All we've done is manipulate the "confidence level." In the first case, it's 95% -- the arbitrary (and "traditional") level generally used in social science research, representing 1 chance in 20 that our sample doesn't correspond to the population. In the second, the confidence level is 99% (1 in 100), and in the third, it's about 68%, or 1 in 3. Don't like your margin of error? We can fix that!*
And none of them are "statistical ties." If you have to have a racing metaphor, they're more and less obscured views of the track. We can't see well enough to know who's ahead, but that doesn't mean the ones we can't see are even.
And then there's:
If Florida has its way, it looks like Giuliani vs. Clinton in a tight race.
Get ready Florida for another nail biter presidential election. (That damn comma of direct address! Out drinking with the hyphens again)
A new St. Petersburg Times/Bay News 9 poll shows America's biggest battleground state is up for grabs by either Republicans or Democrats, and that neither of the front-runners for their party nominations, Rudy Giuliani and Hillary Rodham Clinton, has Florida locked up yet.
I suppose it does. Then again, so do the flight of birds, the chitlins of sheep and the Powerball results (assuming you read 'em right). The poll results are validly and reliably derived, but they can't and don't say anything about what's going to happen in a year (or whether there's any relationship between that eventuality and whether Florida does or doesn't have its say).
Moral? Stick to what you know. Stay on the True Methodological Path and turn not aside. When in doubt, dull it down. If that suggests less (and less prominent) coverage of polling, it should.
* For a price, Ugarte
The nice thing about polling is that it isn't magic. It relies on a finite set of rules, so it's accessible to anybody. And if you attend to the rules (this is the fun part), you'll be right, no matter what the famous and prominent say. Here's a quick summary of the rules. Then we'll see each one broken in a really annoying way:
1) Some methods of sampling public opinion allow you to make reliable generalizations from a sample to the population it represents. These are called "probability" samples. Generalizing from any other sample is the moral equivalent of reading the horoscopes to your colleagues at lunch: If you're lucky, they'll just think you're harmlessly vacant, not dishonest or irresponsible or anything.
2) Generalizing from samples is about likelihood, not proof. (Want proof? Go to seminary.) A competently reported survey must explain how likely it is that the results could have come about by chance (this is the "confidence level") and the width of the band in which your sample is likely to represent the population (the "confidence interval," better known as the "margin of sampling error"). The "margin of error" is not some magical point at which "statistical ties" become untied; anyone who reports it as such is blowing smoke.
3) Survey research, properly done, is a pretty good way of finding out what people think. It's less effective at predicting what people might do, and that utility can fall off in a hurry based on a number of factors. Never get the two mixed up.
Those are some basic guidelines for hanging around in the empirical world, though of course they don't explain everything. The "scientific method" isn't a guide to running your whole life. It's not a good way to figure out what to have for dinner, or whether you like a movie or a piece of music, or whether you've found your True Love. But if you want to talk about a scientific process like survey research, it's the only one you can use. Sorry, no exceptions.
First up, the McClatchy News Service, illustrating Point 1:
Youths favor Obama, Giuliani
Interesting. And how do we support such a claim?
... Those are the headlines from a national survey of 2,526 likely voters ages 18-24 released Wednesday by Harvard University's Institute of Politics.
Hmm. Big sample for a survey. And it's from Harvard!!! If you'll provide the confidence level, we'll be happy to calculate a margin of sampling error, but ... what's that in the back there?
The survey, taken Oct. 28-Nov. 9 by Harris Interactive for Harvard, was conducted online from a sample of young people who agreed in advance to participate. Because of the method, no formal margin of error was attached to the findings.
That's interesting. "Formal" is not a property of confidence intervals (there's no "informal" way of calculating sampling error to contrast to the "formal" one McClatchy is invoking). And there's only one flavor of sampling that doesn't allow this calculation. Wonder if ... gee, do you suppose it's the same methodology Harris ran for Harvard a couple months ago?
This online survey is not based on a probability sample and therefore no theoretical sampling error can be calculated.
You can stop after "not based on a probability sample," because that means we have no idea whether "youths favor" any of the candidates presented. Doesn't matter how big the sample, and doesn't matter how many times you say "Harvard!!!" That's an argument to (false) authority. Arguments to authority are a good way of finding out whether you can stay out after midnight, but they don't affect the rules of probability. Astrology is still astrology if they do it at Harvard!!!
Next up, Howard Kurtz, representing both the mining-town paper in the town where the mine is government (right, that's the Washington Post) and the fallacy of arguing to tradition. Here's the online Q&A:
Gainesville, Va.: Howard, I think it is time to send political reporters and commentators to a remedial course in statistics. You wrote that Obama could as easily be tied with Hillary because his lead of 28 percent to 25 percent is within the margin of error. What the confidence interval means is that if the confidence interval is smaller than the difference between two means we can be 95 percent (not 100 percent) sure that the opinions held in the universe will reflect the higher response to be greater than the lower. If the difference of the two poll results are within the confidence interval, it may mean we can only be sure that one is greater than the other with 80 percent or 75 percent confidence. It doesn't mean they are tied. It really would be helpful if you and your colleagues boned up on this question before you made statements that aren't true.
Howard Kurtz: I stand by what I said. Obama's lead is within the margin of error. In fact, the poll's margin of error was over 4 percent, which makes the 3-point lead even more of what I would regard as a statistical tie.
This isn't another case in which we have a difference of opinion and need to report both sides. Kurtz is wrong, and the reader is right (though "remedial course in statistics" is excessively snarky, and the reader overlooks the importance of applying the confidence interval to both points in the distribution). Tradition -- "that's the way we do it around here" -- is a fine guide to putting up your Xpesmas goodies, but it's not a part of the scientific method.
Put simply, "within the margin of error" and "statistical tie" are meaningless terms, no matter what Howard Kurtz says (and how any tie could be more of a tie than any other tie is a question for the ages). Let's invent a real-life example: We'll survey 800 registered voters in that hotly contested Crook vs. Liar race and see if we can find out who's ahead. We're going to decide in advance how confident we want to be, and based on that, we'll know how likely it is that what we see in the sample represents a real difference in the population. Let's say our result is Crook 51, Liar 47. Who's ahead, if anyone? Let's apply a few margins of sampling error and see.
Start with 3.5 percentage points. The difference is "outside the margin," so called; what does that mean in real life? Discounting the predetermined likelihood of having gotten our results by chance, rather than the degree to which they represent the population, it means Crook is somewhere between 54.5 percent and 47.5 percent, to Liar's range of 43.5 to 50.5. So we could have a perfectly accurate poll that represents a population in which Liar is ahead of Crook, despite what our results say.
All right, let's say the margin of error is 4.5 points -- meaning Crook's lead is "within the margin of error," as Kurtz (and many others) put it. In real life, again, we're X percent confident that Crook's support is between 55.5 and 46.5 percent, to Liar's range of 42.5 to 51.5. Could Liar be ahead in the whole population? Sure. It's less likely than a Crook lead, but evidently it's perfectly possible.
Well then, let's make the margin of error 1.8 points. Aha! In all nonchance cases, Crook's support is above 49 percent and Liar's is below 49 percent. Except for that agreed-in-advance level that we've set for accidents, we got us a lead.
What's the relationship among those three cases? They're all based on the same results. All we've done is manipulate the "confidence level." In the first case, it's 95% -- the arbitrary (and "traditional") level generally used in social science research, representing 1 chance in 20 that our sample doesn't correspond to the population. In the second, the confidence level is 99% (1 in 100), and in the third, it's about 68%, or 1 in 3. Don't like your margin of error? We can fix that!*
And none of them are "statistical ties." If you have to have a racing metaphor, they're more and less obscured views of the track. We can't see well enough to know who's ahead, but that doesn't mean the ones we can't see are even.
And then there's:
If Florida has its way, it looks like Giuliani vs. Clinton in a tight race.
Get ready Florida for another nail biter presidential election. (That damn comma of direct address! Out drinking with the hyphens again)
A new St. Petersburg Times/Bay News 9 poll shows America's biggest battleground state is up for grabs by either Republicans or Democrats, and that neither of the front-runners for their party nominations, Rudy Giuliani and Hillary Rodham Clinton, has Florida locked up yet.
I suppose it does. Then again, so do the flight of birds, the chitlins of sheep and the Powerball results (assuming you read 'em right). The poll results are validly and reliably derived, but they can't and don't say anything about what's going to happen in a year (or whether there's any relationship between that eventuality and whether Florida does or doesn't have its say).
Moral? Stick to what you know. Stay on the True Methodological Path and turn not aside. When in doubt, dull it down. If that suggests less (and less prominent) coverage of polling, it should.
* For a price, Ugarte
Labels: polls
posted by fev at 7:45 PM
0 Comments:
Post a Comment
<< Home