More polling delusions
Here's the flip side of newspapers' annoying obsession with insignificant fluctuations in public opinion surveys: the idea that all polls are fibs. It's worth noting because, from the editor's perspective, it underscores the same point. Polls aren't magic, but they're only useful if you know what they do and how they do it.
Today's entry, from a regular Sunday column in one of the two leading dailies:
The hundreds of polls floating around don't help me decide. Like statistics, polls can be spun to make any point the pollster is pushing -- it's all in the way the questions are posed.
In a word, no. It's not "all in the way the questions are posed." That's not to say it can't be. If you ask a question like "Did you know that my opponent enjoys congress with domestic pets and barnyard animals?" of a proper random sample and 90 percent say "no," you can report that "90 percent of Missourians don't know that my opponent enjoys congress with domestic pets and barnyard animals!" And you can even report a margin of sampling error!*
The way to make sure it's not "all in the questions" is to be sure the deck isn't stacked. Don't load presuppositions into yes-and-no questions. Make sure scale values are balanced. Make sure semantic differentials are different ("competent" to "incompetent," not "brilliant" to "competent"). Stick to questions that can't be spun: Did you vote in the last election? Are you registered to vote? Do you plan to vote in this election? Who do you plan to vote for: A, B, other, undecided?
When we say "polling isn't magic," that's exactly what we mean. Not "polling is bad magic," but "polling is mechanics." Maybe if we show people more of what goes on behind the curtain, they -- and we -- will be less obsessed with the damn things. (And no, by the way, polls aren't supposed to "help me decide.")
* For our J4950 customers: Name a formula that you think might make a return appearance on the next exam. What statistic represented there by "1.96" needs to be in every poll story,** without exception?***
** This doesn't include things like the 1A Thursday centerpiece. Fake polls don't have statistics.
*** Bonus: Would the margin of sampling error for our hypothetical question be larger or smaller if 50% had said "no"? Explain.
Today's entry, from a regular Sunday column in one of the two leading dailies:
The hundreds of polls floating around don't help me decide. Like statistics, polls can be spun to make any point the pollster is pushing -- it's all in the way the questions are posed.
In a word, no. It's not "all in the way the questions are posed." That's not to say it can't be. If you ask a question like "Did you know that my opponent enjoys congress with domestic pets and barnyard animals?" of a proper random sample and 90 percent say "no," you can report that "90 percent of Missourians don't know that my opponent enjoys congress with domestic pets and barnyard animals!" And you can even report a margin of sampling error!*
The way to make sure it's not "all in the questions" is to be sure the deck isn't stacked. Don't load presuppositions into yes-and-no questions. Make sure scale values are balanced. Make sure semantic differentials are different ("competent" to "incompetent," not "brilliant" to "competent"). Stick to questions that can't be spun: Did you vote in the last election? Are you registered to vote? Do you plan to vote in this election? Who do you plan to vote for: A, B, other, undecided?
When we say "polling isn't magic," that's exactly what we mean. Not "polling is bad magic," but "polling is mechanics." Maybe if we show people more of what goes on behind the curtain, they -- and we -- will be less obsessed with the damn things. (And no, by the way, polls aren't supposed to "help me decide.")
* For our J4950 customers: Name a formula that you think might make a return appearance on the next exam. What statistic represented there by "1.96" needs to be in every poll story,** without exception?***
** This doesn't include things like the 1A Thursday centerpiece. Fake polls don't have statistics.
*** Bonus: Would the margin of sampling error for our hypothetical question be larger or smaller if 50% had said "no"? Explain.
posted by fev at 10:27 AM
3 Comments:
The day you get t-stats regularly mentioned in polling stories is the day I might consider returning to the “news” business. But we both know it ain't gonna happen.
Speaking of recent news, please remind your stoodnts that porn stars and framed photos are hung.
People are hanged.
It's in the style book, right alongside that other unread page on firearms.
Small steps, gentlemen. Small steps.
First, it's confidence level and confidence interval in every poll story. Then it's a ban on featurized science ledes. Then it's test values and critical values of t and F ratios in stories. Then your pitiful, trusting newsroom falls into our hands like an OVERRIPE FRUIT!
Post a Comment
<< Home