Wednesday, April 27, 2011

Real poll, bogus conclusion

"Correct me if I'm wrong," asks the Chief Testy Copy Editor (and when he speaks, rimrats listen), "but there's nothing in this poll that says Obama is less popular because of gasoline prices."

Short answer, of course, he's right. But it's never too early for a little ranting about public opinion surveys and their interpretation. What the Post is doing is fabricating a causal relationship from coincidental data, because ... well, why not?

Soaring gasoline prices are biting into household incomes and nibbling at Americans’ fuel consumption — and support for President Obama, according to a Washington Post-ABC News poll.

About six in 10 respondents said they had cut back on driving because of rising fuel prices, and seven in 10 said that high pump prices are causing financial hardship.

Obama, like previous presidents in times of high oil prices, is taking a hit. Only 39 percent of those who call gas prices a “serious financial hardship” approve of the way he is doing his job, and 33 percent of them say he’s doing a good job on the economy.

I'd class this as one of the most pervasive of ideological media biases. It's not a partisan bias; the Post is equal-opportunity stupid* with survey data (and, in general, pretty drooly toward Obama). But it's distinctly ideological; it reflects a set of assumptions about how the world ought to be, and one of those assumptions is that Expert Journalists are supposed to find the political meaning in a story, whether it has a political meaning or not.

Now, it is possible to get an idea of whether changes in consumer prices are affecting public perceptions of executive performance. But you do it by asking ("Have gasoline prices affected your opinion about whether the president is doing a good job?"), not by making stuff up. Which is the problem implied in this graf:

In a hypothetical matchup with former Massachusetts governor Mitt Romney, the top GOP performer in the Post-ABC poll, Romney wins by 24 points among the independents who have taken a severe financial hit because of gas prices, and the president is up 7 percentage points among other independents.

Our comparison -- and there's a lovely chart to prove it! -- involves different kinds of independent voters, divided on how they describe the sort of "financial hardship" gasoline prices have caused in their household: serious, not serious, or no hardship (collapsed for analysis into "serious" and "other," which is fine). And the survey finds a real difference: independents who say they've been seriously affected are significantly more likely to disapprove of Obama's performance that independents in the "other" category. (The Post doesn't bother to report the margin of sampling error for independents,** which is unfortunate, because the confidence interval for the bogus conclusion in the hed is far more relevant than the alleged mean price at which people would allegedly change their driving habits.)

The problem, of course, is that we have no idea what's the chicken and what's the egg -- or even if there's some antecedent chicken laying both eggs. Are independents who report a serious hardship predisposed to dislike Obama, or do they perceive "hardship" differently because they're still waiting on the birf certificate, or has their attitude actually changed because of the change in gasoline prices? We don't know. And that's as far as the hed can go.

I don't think the Post can plausibly be considered "anti-Obama," though the ideological biases in Tuesday's edition do seem to redound against the president. The first deck on the 1A lede story ("Syria escalates lethal crackdown") says "Pressure rises on Obama," which is more or less entirely made up. Put more politely, we could say it reflects a strong exceptionalist and anti-realist bias -- a refusal to sit back and say, yeah, there are weddings that we don't have to be the bride at, and this is one.

I think Ezra Klein is a moron for declaring that "America is mired in three wars," but that's not because I feel some need to defend Obama's bumbling in Libya; it's because people who declaim about militarized disputes without being able to distinguish different levels of them are generally morons. That's one of my ideological biases. I'm anti-exceptionalism and pro-realism, and if you don't like it, you can go get your grammar advice from the unquiet ghost of Bill Safire.

And I need to add this in conclusion. The Post screwed up out of professional ideology. It ran a bogus hed and story, but that puts it in league with other papers that mistakenly assume "news judgment" is a substitute for reading the questions and looking at the numbers. Don't mistake this for the more deliberate and better organized lies on offer at the Fair 'n' Balanced Network.

* To be fair, you can -- and should -- say this about many if not most grownup news organizations.
** Judging from the poll data, it's around 4.8 points at 95% confidence, but it's really rude to make people calculate their own ERAs, don't you think?



Anonymous Anonymous said...

And yet, these are the building blocks from which so much discourse (and even economic activity) is built.

Be thankful you don't spend much effort criticizing media reporting: the sample sizes for your local radio ratings, for example, are small enough to put the entire top 25 within the 95% CI -- yet the industry and the advertisers agree to spend many millions of dollars annually on the basis of these fictitious numbers because it would cost too much to collect meaningful ones. (And that's just for the 12+ numbers: the real money demos are males 18-34 and females 25-54, and each of those subsamples may have as few as a hundred respondents depending on the size of the market.)

2:45 AM, April 27, 2011  

Post a Comment

Links to this post:

Create a Link

<< Home