Bad hed, ray of hope, bad story

Time to complain about one of the Worst of All Hed Adjectives for a moment. That word is "faithful." Lots of writers use it in the sense of "having religious faith" -- not wrong, though Webster suggests it's obsolete. The biggest trouble with it in heds like the one shown here is other meanings are equally, if not more, accessible. Are the cousins down at the Dispatch talking about people who are conscientious about voting or religious people who vote?

Turns out it's the latter, which leads to another problem. The survey described under the hed doesn't attribute the toe-keeping characteristic to all religious voters, which suggests that the paper is adopting some of the pollster's baggage. That'd be the Barna Group, which calls for a few comments on the art of polling that may be worth noting over the next few months.

There's a common misperception out there that pollsters can wave a secret magic wand and make polls say anything they (or their paymasters) want. That's not true -- or at least not very true. Given a properly drawn random sample and mildly professional execution, a question like "If the election were held today, would you vote for A, B or someone else?" is going to produce a (more or less) valid and reliable result, no matter who asks it.

That's not to say there aren't evils done in the name of polling. One form arises from questions that are badly written or tendentious. Here's a genuinely stupid one from a Fox poll last month:

When talking about United States citizens of different ethnic or national descents, do you think the word American should come before or after the descent? That is, should Americans of Irish descent be called Irish-American or American-Irish, and should Americans of African descent be called African-American or American-African?

Does this mean Fox's results on the "If the election were held today ..." question aren't an accurate representation of what registered voters think?* No; it means Fox's polling firm will throw in anything the bottom-feeders at Fox think needs to be asked.

The other big offense that "pollsters" (as opposed to the journalists who invent portentous trends based on perfectly innocent results) commit is not showing their work -- in this case, describing outcomes without demonstrating how thoroughly those outcomes are supported. Such is the case with the poll the Dispatch falls for here:

If the presidential election were today, Barack Obama likely would become the first Democrat in 32 years to win among born-again Christians.

In fact, of the 18 faith segments tracked by pollster George Barna, the Illinois senator currently leads in all but three. Only two groups of evangelical Christians and those with an "active faith" favor Republican John McCain. (The writer means two groups, evangelicals and actives, not two groups of each. Yoo hoo! Editors!)

But Barna finds that the political landscape among the faithful is churning rapidly, with support for Obama dropping almost across the board.

Time to stop the tape and unpack a few things. Yes, your first questions at this point should be about sample size: How big is the poll, how big are those subgroups? Without that information, you can't make a reliable judgment about whether the "political landscape" is "churning." (That's just a mixed metaphor; "churning rapidly," on the other hand, is an invention on the reporter's part. "Rapid" isn't the sort of thing you can measure with two polls two months apart.)

There's where Barna loads up the old spitball -- which, given that it's a free country and all, he's welcome to do. But it's the paper's job to ask the questions, and it fails. For one, according to the graphics, the overall sample is 1,003 adults, but the conclusions Barna reports are among "likely voters." In the June poll, that was a substantially smaller chunk, or 561 people. That makes for a noticeable difference in the confidence interval. Does the August poll make the same distinction? Gee. We don't seem to say.

Now, how big are those 18 "faith segments"? No idea. How many of them overlap? A bunch. How can we assess that "across the board" claim? Apparently through faith alone and not by works, and that isn't going to get you into methodology heaven.

The Barna Group doesn't make any secret of its biases, at least. And the biases don't mean the raw data are bad. They do, though, strongly suggest that any conclusions about "evangelicals" should be taken with a barrel of salt. Broadly, one of the ways Barna sorts Christianity is into three categories (look up the full definitions toward the end here): born-again, evangelical and "notional." So when the pollster himself offers assertions on the order of "While there is still a decided preference for Senator Obama, the more conservative element of the Christian population is slowly coming to grips with what an Obama presidency might be like," the writer needs to be aware -- and to make the audience aware -- that things like "core" and "conservative" might mean very different things in Barna World than they mean in the corporeal one in which political campaigns are fought.

You can say one good thing on the hed's behalf. It doesn't make any proclamations about who's ahead or behind. More survey heds should be so inconclusive.

* From the list of questions (give Fox credit for linking to the whole thing), it also appears that the preference question comes first -- meaning, at least, that the results aren't misprimed by questions like "How likely do you think it is that Obama is secretly a radical terrorist? Verry likely, somewhat likely, or only a little likely?"