Objectivity failure
Time for another brief rant about the public interpretation of survey research. (If you're growing tired of those, may we recommend the excellent Comics section downpage?)* As we mentioned at the weekend, there's a widely held -- and generally wrong -- belief that some people called "pollsters" do evil magic stuff to numbers while we aren't looking and thus distort public opinion to reflect (a) their own Maoist leanings, (b) the fascist tendencies of their paymasters, or (c) either of the above, as long as they can embarrass innocent citizens in the process.**
That'd actually be a really stupid way to do business. Polls can, and do, ask vacant, loaded, double-headed, misnegated or openly racist questions. But poll questionnaires tend to sound alike on the relevant stuff:
If the election were held today, would you ...
Would you say that's very likely, somewhat likely, or ...
On a scale of 1 to 5, with 1 being "very happy" and 5 being "very unhappy," how would you ...
because that's how you get reliable answers. You could draw a random sample of 1,004 people and ask if they're planning to vote for the terrorist or the war hero, but if that's your goal, it's a lot cheaper to just make up a few percentages. Properly sampled and professionally arranged, a poll from Fox is going to be every bit as reliable on the big-league questions as one from -- well, if you can imagine a left-wing U.S. news organization, a poll run by a left-wing news organization.
Which gets us to "objectivity" -- that happy state in which we base our results on things as they work in the real world. What "objective" should mean is that everything's in the open: questions, order, results, subgroup sizes, all that. If you want to run all the tests again and see if anybody's cheating, you can. We don't have to make dark inferences about motives; we can talk about facts.
Does that give you the idea that "objectivity" isn't, or shouldn't be, a requirement to let people tell lies? Hope so, because that's the point of today's example. McClatchy is reporting (it's a bit hyperbolic, as usual) on a survey of registered female voters that -- surprise, surprise -- finds a pretty massive lead for Obama among the so-called GenY crowd. Calling forth this response:
EMILY's List overwhelmingly supports Democratic candidates, and that slant is going to figure into its numbers, said Wendy Wright, the president of Concerned Women For America, a political action group based in Washington that opposes legal abortion.
Erk. This is a genuine failure of objectivity. Wendy Wright is welcome to say or think what she wants, but the reporter (failing that, the desk) is justified here in telling her to sit down and shut up. She can say she doesn't like the numbers, or that she expects American womanhood to come to its senses soon, but when she says people who make their living by running polls are stacking the deck on their results, she's out of bounds. (Try that with accountants or lawyers and they're likely to sue you for libel.) "Objectivity" gives her the information to run a survey of a random sample of 1,400 female voters and see whether the results differ significantly from the ones reported here. It doesn't create a right to lie.
But don't the numbers need some sort of comment from people who don't like Emily's List? That's actually a better question. If the numbers are accurate, why does it? And if they aren't, why is the story running?
* And yes, this is an editing issue. Editors -- until some front-office genius decides to put us out of the sport altogether -- are still the folks who are empowered to stand up and ask things like "Why are we putting that crap in the paper?"
** Response biases are really interesting. Lying about voting behavior tends to go in only one direction: People don't say "I didn't vote" if they did vote, but they're happy to say "I voted" when they didn't.
That'd actually be a really stupid way to do business. Polls can, and do, ask vacant, loaded, double-headed, misnegated or openly racist questions. But poll questionnaires tend to sound alike on the relevant stuff:
If the election were held today, would you ...
Would you say that's very likely, somewhat likely, or ...
On a scale of 1 to 5, with 1 being "very happy" and 5 being "very unhappy," how would you ...
because that's how you get reliable answers. You could draw a random sample of 1,004 people and ask if they're planning to vote for the terrorist or the war hero, but if that's your goal, it's a lot cheaper to just make up a few percentages. Properly sampled and professionally arranged, a poll from Fox is going to be every bit as reliable on the big-league questions as one from -- well, if you can imagine a left-wing U.S. news organization, a poll run by a left-wing news organization.
Which gets us to "objectivity" -- that happy state in which we base our results on things as they work in the real world. What "objective" should mean is that everything's in the open: questions, order, results, subgroup sizes, all that. If you want to run all the tests again and see if anybody's cheating, you can. We don't have to make dark inferences about motives; we can talk about facts.
Does that give you the idea that "objectivity" isn't, or shouldn't be, a requirement to let people tell lies? Hope so, because that's the point of today's example. McClatchy is reporting (it's a bit hyperbolic, as usual) on a survey of registered female voters that -- surprise, surprise -- finds a pretty massive lead for Obama among the so-called GenY crowd. Calling forth this response:
EMILY's List overwhelmingly supports Democratic candidates, and that slant is going to figure into its numbers, said Wendy Wright, the president of Concerned Women For America, a political action group based in Washington that opposes legal abortion.
Erk. This is a genuine failure of objectivity. Wendy Wright is welcome to say or think what she wants, but the reporter (failing that, the desk) is justified here in telling her to sit down and shut up. She can say she doesn't like the numbers, or that she expects American womanhood to come to its senses soon, but when she says people who make their living by running polls are stacking the deck on their results, she's out of bounds. (Try that with accountants or lawyers and they're likely to sue you for libel.) "Objectivity" gives her the information to run a survey of a random sample of 1,400 female voters and see whether the results differ significantly from the ones reported here. It doesn't create a right to lie.
But don't the numbers need some sort of comment from people who don't like Emily's List? That's actually a better question. If the numbers are accurate, why does it? And if they aren't, why is the story running?
* And yes, this is an editing issue. Editors -- until some front-office genius decides to put us out of the sport altogether -- are still the folks who are empowered to stand up and ask things like "Why are we putting that crap in the paper?"
** Response biases are really interesting. Lying about voting behavior tends to go in only one direction: People don't say "I didn't vote" if they did vote, but they're happy to say "I voted" when they didn't.
Labels: polls objectivity
posted by fev at 10:12 PM
3 Comments:
Speaking of things you've all heard me say: polls aren't news, and should be placed on the same page as the horoscopes.
Polls are better than that, but they're at best a snapshot of what people think right now.
Still, I REALLY HATE this obsession with finding someone who thinks differently and quoting them, as if they matter. Sun rises in east, and Billy Jones, a Flat Earther, says that of course people who think the earth is round would say that.
Well ... procrastination mode fully on here, so, I'll agree a little and expand a little.
The stuff that makes polls valid is the stuff that makes them "scientific" -- to oversimplify, that they're probabilistic, tentative and incremental. That doesn't mean polls don't tell you important things about public opinion; it means they almost never do so in a way that comports with news values. News likes to be definite and conclusive.
As a rule, if a poll result is exciting enough to make a 2/60/3, somebody screwed up.
Post a Comment
<< Home