Pollwise, these are trying times
Q: Are there worse things you can do with polling data than a little creative bending to fit the story line?A: Yep. Indeed, several particularly unpleasant parts of hell* are reserved for those who violate the First Commandment of survey research: No generalizing from nonprobability samples, ever. Results of any self-selecting poll have exactly the statistical validity of the horoscope and may be appropriately displayed on the comics page.
Real news organizations don't do stuff like this, so it's a good thing we have Fox to keep the editing classroom supplied with awful examples. For the record, when a few thousand of Rush Limbaugh's closest friends call in to complain that the in-the-tank media are suppressing this valuable bit of data:
None of this is true. The troops have not been polled. There is no "landslide of support" for anyone, because we have no way of knowing what "the troops" think. Military Times had the good sense to acknowledge that its readership doesn't accurately represent "the military"; the MT lede refers to the military's "professional core," and the third graf is more detailed: "The group surveyed is older, more senior in rank and less ethnically diverse than the overall armed services." Fox ignores those concerns, which conflict with its narrative:
A poll by the Military Times newspaper group suggests that there is overwhelming support for John McCain among U.S. troops in every branch of the armed forces by a nearly 3-1 margin.
According to the poll, 68 percent of active-duty and retired servicemen and women support McCain, while 23 percent support Barack Obama. The numbers are nearly identical among officers and enlisted troops.
But in either case, we're still missing the bigger point: Respondents selected themselves into the survey. That creates an uncontrollable, nonquantifiable source of error. No one has any way of knowing whether the volunteers accurately represent the readership.
As is often the case, we really can't tell whether Fox is being unusually dishonest or just unusually stupid. Since those conditions aren't exclusive, it might well be both.
* Lots of mosquitoes, no copy editors, nothing on tap but Bud Light.
posted by fev at 12:00 PM
2 Comments:
This on the same day my morning paper says "Military families tilt to Obama." The Toronto Star here is not reporting on a poll at all; the story is mostly about Blue Star Families for Obama. But it does include this sentence:
"In a forthcoming article in The New Republic, infantry officer and author Jason Dempsey says while studies of military politics often show a tilt to the right, they tend to focus on 'white, older and more senior' officers who make up only 14 per cent of army personnel."
Apropos: "20 questions a journalist should ask about a poll"
The only quibble I have with the list is that "who paid for this poll" should be No. 1, not No. 2.
Otherwise the questions are pretty much the ones we've been hashing out over the years (and have been generally ignored).
And now for Strayhorn's Law: Polls are not news, and should be on the same page with the horoscope.
Post a Comment
<< Home