Lying about lying about statistics

If you're a Usual Suspect or other regular reader, you've probably noticed that we spend a lot of time complaining about quantitative survey research. Let me offer a general proposition to put that into perspective: You can complain about anything in a poll except the results.

It's legitimate to attack dishonest methods, stupid interpretation, biased discussion, or whatever. But if a poll has been conducted according to a few basic professional standards, you can't blame a result you don't like on the alleged biases of the people who reported the research. That's just another way of lying with statistics, and to support that assertion, we're going to introduce somebody else who complains about surveys: L. Brent Bozell III of the Media Research Center:

It has become almost amusing, watching how the so-called “news” media are manipulating their own polls to keep the political weather sunny for their hero. The Washington Post kicked off President Barack Obama’s European trip with the headline “Blame For Downturn Not Fixed on Obama.” Of course, what was “fixed” was the poll itself.

They did the usual tricks for a more liberal sample of “public opinion” – they polled on the weekend and oversampled Democrats (36 percent Democrat, 25 percent Republican). By themselves, these things are shameless – but expected.

Let's stop the tape in mid-graf for a moment. A big assertion -- there's a set of "usual tricks" employed to get liberal results in polls -- is followed by two specific ones the Post is alleged to have indulged in. Two minor problems are worth noting: Those "tricks" don't produce a "more liberal sample," and the Post didn't do either of them.

As a rule, if you're going to lie, you should lie about stuff that's hard to check, not stuff that's easy to check. The story and the breakout available online both say the poll was in the field for four days, Thursday through Sunday. (Maybe that's the weekend at the Media Resource Center; nice work if you can get it.) That's twice as long as Opinion Dynamics stays in the field when it polls for Fox, but that doesn't make two-day surveys a trick for finding a conservative sample. Other things equal, a shorter poll is probably more susceptible to error* than a longer one. If you're in the field Friday and Saturday, you might get a less representative sample on age, and that could bias the results if your issue is one on which younger people are notably more conservative or liberal. But I can't imagine why polling on weekends only would reliably produce a more liberal sample,** and at any rate, that's not what the Post did. C'mon, L. Brent: If you're going to make stuff up, at least make us work at it a little.

The next accusation is more serious and a bit more sinister. He's saying the Post is lying about methods. It's reporting a random sample, but it really went looking for extra Democrats to get the results it wanted. That's pretty serious; stupidity isn't a firing offense, but research misconduct is. Conveniently, it looks as if this assertion too is entirely made up.

First, party identification isn't gender. It isn't evenly balanced in the population, and if you expect it to be, you aren't paying attention. Fox/Opinion Dynamics found an 11-point difference in November '07 and a 42%-33% result in July. So it seems safe to conclude that significantly more people identify as Democrats than as Republicans these days, no matter whether Fox or the Washington Post asks them. Second, there is a technique called oversampling, and legitimate surveys sometimes use it, but they don't use it to skew proportions in the overall sample. (If you want to do that, it's easier and cheaper to just make the numbers up.) They use it to make subgroup results tighter.*** If you're going to accuse people of a research crime, L. Brent, do them the courtesy of getting the crime right.

And still that wasn’t enough of a slant. Check out the way this question was asked by the Post pollsters.

“How much of the blame do you think [fill in the blank] deserves for the country’s economic situation?” The choices were corporations, banks, consumers, the Bush team, and the Obama administration. There’s a built-in pro-Obama bias in there already: assigning blame to Obama for the current economy when he’s been in office for nine weeks just seems harsh to most people.

Whether or not something "just seems harsh" to "most people" is an empirical question. You can measure it, which means "well, that's how it seems to me" isn't a very useful form of evidence. And given the level of invective from the droolers on the topic over the past few months, it seems this is a public-opinion question worth exploring: How many people agree with the Fox/Limbaugh narrative about the "Obama recession"?

You might have concluded by now that L. Brent Bozell III is a liar and a buffoon,**** and you'd be right. In a way, that's unfortunate, because he's about to stumble on a fairly important point, though he doesn't seem to know why:

These are fair descriptions, I think we can say. But now check how they identified the problem when it was a politician: Should the public blame Bush for “inadequate regulation of the financial industry”? Or is Obama to blame for “not doing enough to turn the economy around”?

What kind of left-wing pollster wrote these questions? Is Obama “not doing enough”? We’re being buried in trillion-dollar Obama proposals and he should be faulted for “not doing enough”? How about the crazy idea that maybe, just maybe, he’s doing too much? This question makes sense only if the goal is to assist Obama politically.

Here are the choices the Post poll offered. Raise your hand when you see the problem:
a. Banks and other financial institutions, for taking unnecessary risks
b. The Bush administration, for inadequate regulation of the financial industry
c. Large business corporations, for poor management decisions
d. Consumers, for taking on too much debt
e. The Obama administration, for not doing enough to turn the economy around

Bozell's point is that there's no response for his (and Limbaugh's, and Fox's) belief that it's Obama's fault; the country is panicking as the colored fella sheds his civil veneer and reveals the heathen commie rat beneath. And he's right (about the lack of a category for his preference, that is). In generalizable terms, two-headed questions like these are methodologically inept because they aren't exclusive and exhaustive. Which one do you pick if you blame Obama for being a socialist, or Bush for screwing up everything else about the country, or Reagan for setting the country on the destructive course his acolytes pine for today?

A better way to handle this one -- drawn from the long list of advice that's worth every cent the Post has paid for it -- is to separate the questions about which agents are to blame (Bush, business, banks, consumers, Obama) and which characteristics are to blame (deregulation, incompetence, greed, stupidity, Stalinism). That could tell you something interesting about an important phenomenon -- how many people are taking what they hear from the Beck/Hannity/Limbaugh axis to heart? -- without leaving an enormous mass of error hanging around the center of the question.

I'm not prepared to call it a clear-cut example of ideological bias on the Post's part. Not that the Post doesn't have ideological biases, which it does, but that this one looks more like basic cluelessness. Even asked correctly, what would make a story like "two months into the new administration, most people don't blame the new guy" worth 1A play and a hed?

That's too bad, because when the poll sticks to the basics, it actually throws off some interesting results:

The number of Americans who believe that the nation is headed in the right direction has roughly tripled since Barack Obama's election, and the public overwhelmingly blames the excesses of the financial industry, rather than the new president, for turmoil in the economy, according to a new Washington Post-ABC News poll.

The "right direction" result is pretty striking, making it even more regrettable that the Post put such weight on the sloppily worded blame question. (For the record, the public "blames" big business as much as it blames the bankers, and consumers slightly but not significantly more than the Bush administration.) Thus, it's harder to complain about all the manifest lies and biases in Bozell's critique, and there are quite a few:

So Obama’s trying to implement socialism at 120 miles per hour, and with a straight face, the Post reported that 62 percent of those surveyed still see Obama as a “new-style Democrat who will be careful with the public’s money,” while 32 percent see him as an “old-style tax-and-spend Democrat.” An accurate assessment by the Post would conclude that a) Obama’s accelerated socialist policies make most conservatives pine for the good old days of “tax-and-spend Democrats” and b) 62 percent of the public has no idea what is going on in Washington – primarily because they rely on outlets like the Post for their “news.” (Here, Bozell is not complaining that the newspaper made things up; he's complaining that it didn't make up the sorts of things he wanted somebody to make up.)

Then there were poll questions that the Post editors didn’t want on the front page – or even anywhere in the poll story by political reporter Dan Balz and pollster Jon Cohen. On the front page, Post readers saw the big news – a bar graph showing that 60 percent approve of how Obama is handling the economy. But if you look at the Internet and read the actual poll, there’s another number the Post deliberately left out. Pollsters asked “Do you approve or disapprove of the federal government's overall response to the economic situation?” Forty-nine percent said they supported the overall federal government response.

So who, boys and girls, is the “federal government? It’s controlled by a Democratic president, and a strongly Democratic Congress. One could clearly state, then, that less than half of the public supports President Obama’s economic agenda. But the Post ignored this so as to trumpet the opposite.

Yes, one could state that. And, in a way, one could state that the Post "ignored" it. On the other hand, the Post also ignored the doubling in the proportion approving of the federal government's overall response since mid-December (when, boys and girls, the president was a "Republican"). "Strongly approve" has quadrupled. It's a bit difficult for a grownup to conclude that the Post is selectively presenting evidence to make Obama look good if it left this part out.

Clowns like L. Brent Bozell III make it hard on the rest of us. When the Post (or the NYT) hears that somebody's bitching about its poll reporting, it has every reason to conclude in advance that the complainant is a paid liar who wouldn't know a confidence level if it jumped up and bit him (or her) in the arse. If you, in your best indoor voice, want to raise a question about instrument design or the appropriate interpretation of results at the fuzzy edge of the confidence interval, the Post is likely to mistake you for the former. That's not fun for you, and it's not much help for the Post either.

Good reporting about the press is like good reporting about polls: Know the rules, answer to the evidence, describe what you see rather than what you want to see. If L. Brent and the gang over at the Media Research Center ever figure that out, they might have something useful to say.

* Meaning, simply, variation you can't account for by stuff you can measure or control.
** If you know of any lit to the contrary, drop me a note, OK?
*** Quick example. Say 10% of the population has some characteristic (ethnicity, home language, whatever) of interest. An off-the-shelf sample size of 800 yields a confidence interval ("margin of sampling error") of 3.5 points at 95% confidence. For your minority, though, it's over 11 points -- so a 50% result in the sample could accurately reflect a value anywhere from 39% to 61% in the whole population. If you keep sampling until you have 200 respondents from the minority, the confidence interval is down to around 7 points, and you can make more precise comparisons about subgroup preferences. When you talk about the whole sample, though, you weight the subgroup response to reflect the subgroup's proportion in the population. Make sense? This might be on the final.
**** Hey, I said stupidity and dishonesty were different phenomena; I didn't say they couldn't co-occur.