Democrats eat babies, too

What are our late-night-news-hungry friends finding over at the Web site of the Fair 'n' Balanced Network? Glad you asked!

Certainly seems to be a particularly interesting story there, doesn't it? Those pesky Democrats want us to lose the war! And it's a real poll! So it has to be true!

Well, let's cavil about execution a bit before we get to substance. Give Fox a point for posting a PDF of the questions -- well, at least all the questions starting with No. 40. You have to wonder what might have lurked, say, at 27:
Yur leg it has a flavr:
[] Strongly agree
[] agree
[] neutral
[] disagree
[] strongly disagree

And methodologists are divided on the mythical Question 39:
[] Ya! Cthulhu fthagn!
[] No! Cthulhu no fthagn!
[] Neither ya nor no!
[] Not sure!
[]Other!

But let's talk about what we can actually see here in the empirical world, not what might have primed the respondents during the first nine-tenths of the survey or anything.

Fox reports the poll's margin of sampling error* as 3 points, which is understated; it's about 3.3 points, or 10 percent larger than we're told (that's a maximum, for the entire sample on a question with a 50-50 split; for Democrats on the question at hand, which is split about 4-1, the margin is 4.1 points). The poll was conducted over two days, in the evenings, which is a good way to lose points in your beginning methods class. This claims to be a national sample of registered voters, but it's a national sample of registered voters who are home in the evening. In this zip code at least, such a sample is a long way from being representative.

In fairness, you might respond here: Who cares? One major party is defending America and one wants us to start speaking German (oops! Arabic!), and you're bitching about stuff like basic standards in sampling and reporting public opinion? Well, yeah. Standards are how we know that research results are valid and reliable. And another couple are worth mentioning.

The reason you always want to introduce the concept described in graf 4 as "margin of sampling error," rather than "margin of error," is that "sampling" is only one of the many kinds of error that bedevil research. It's an annoying cliche to say "there are no stupid questions," because in fact there are rather a lot of them. Stupid questions are a leading source of error too, as in these samples from other recent Fox/Opinion Dynamics surveys:

Which of the following two descriptions of the way things are today do you agree with more?...The nation's economy is not improving, Osama bin Laden has not been captured and the United States is in an unnecessary war with Iraq. The nation's economy is strong, there have been no new terrorist attacks in the United States and the world is safer with Saddam Hussein out of power and an elected government in Iraq. (May 2006)
Each of the options has three components. Which one is the poor respondent responding to? What other ways are there of describing "the way things are today"?

How do you think President Bush should decide how to proceed in Iraq--should he follow his gut instincts about what is right or follow public opinion polls? (July 2007)
False dichotomies are almost by definition Stupid Questions. If you were writing this question, can you think of a few other options open to the nation's chief executive?

Who do you think is more to blame for the way things are going in Iraq?...The Bush administration for starting the war, Congress for approving the war and continuing to fund it, the Iraqi people themselves for failing to take control, Al Qaeda and insurgents for attacking the region ... (July 2007)
Check out the embedded presumptions in the available responses. Choice 2 assumes that there was a vote in Congress to "approve" "the war," which there wasn't, and it double-barrels the initial vote with an unspecified sequence of later votes that may or may not have specifically been about "the war." Choice 3 assumes both that the "Iraqi people" could have done something called "taking control" and that they tried and failed to do it. Choice 4 drags a large red herring across the trail by dragging Qaida into the mix.

So "question error" broadly is the set of stuff that asks: Given that N% (+/- 3.1) of subjects said "yes" or "1" or "2," what does "yes" or "1" or "2" really mean? Which, with all due respect, is how rational grownups need to approach Fox's Question 40:

Do you personally think the world would be better off if the United States loses the war in Iraq?

OK, how would you have responded? "Gee, FOX News, I thought we'd already won that war!" Or: "You 'win' a war when you plant your flag in the smoking ruins of the enemy capital. Do you mean 'would the world be better off if the United States pulled up its flag and left Baghdad,' or do you mean 'would the world be better off if Iraq planted its flag in Washington'? Because those are two radically unrelated questions that have somewhere between 'lots' and 'nothing' to do with the political choices I might decide on?"

Polling isn't inherently dishonest, but dishonest polling is inherently dishonest (and, yeah, ineptitude doesn't exactly help the situation, either). So I suppose this is a reminder that the great battle isn't between "left" and "right" wings of American journalism. It's between people who try pretty hard to get stuff right (and often, for whatever reasons, fail -- sometimes spectacularly) and paid professional liars, which would be the side Fox is on. Do bear that in mind as the season wears on.

Still wondering? Check out the last item under "Latest News," above: "Barack Obama Stops Wearing American Flag Lapel Pin." Q-E-sodding-D.

* I'm getting tired of saying this, but what the heck: "Margin of error" is meaningless unless you report the confidence level it's calculated at. Report your confidence levels or be branded a goober for life.