On polls and reading them

It's Public Opinion Week in your friendly neighbor- hood political reporting class, and as usual the examples keep pouring in. Let's look at a few things not to do with survey research, on the off chance it'll sink in next time the AME for presentation is demanding a big, conclusive graphic to go with your poll story.

That's not the problem with the front shown here; the 1A clip is just a reminder to the Columbus gang to (ahem) cut it out permanently and forever with using roller-coaster photos to illustrate thumbsuckers about the economy. We've been over this before. Stop it.

The lede hed -- "Ohio voters lean GOP once again" -- isn't really the problem either (though the "once again" isn't very well supported, and it probably doesn't take a lot for some readers to recall the D's longstanding reputation as a tool of the Republican Party,* and you can go from there). The problem is with the lede, which illustrates a basic mistake you should never make with survey data: Pretending your numbers reflect some purported trend you haven't measured. Here we go:

The Massachusetts revolution might be spreading to Ohio. (And then again, it might not! That's why -- even if we could define the "Massachusetts revolution" and had some claim to having measured it -- "might" ledes are on the forbidden list.)

Although the vast majority of Ohioans don't blame Democratic Gov. Ted Strickland for the state's economic woes, they favor GOP challenger John Kasich by 6 points in this year's gubernatorial election. (Funny, I didn't get the idea that the Massachusetts "revolution" could be summed up as "We don't blame the incumbent, but we're leaning toward a nine-term congressman who hung around on Fox a lot after leaving office." A safer conclusion based on the limited data available -- kids, don't go around casually comparing polls conducted 11 and 10 months before the election by different agencies -- might be that Strickland appears to have picked up a little more support than Kasich recently among people who might be making up their minds. And just to be strict: Never draw conclusions about what "Ohioans" think from a sample of likely Ohio voters. Don't generalize to a population you didn't sample. That's a rule. Do not break it. Ever.)

The result is fueled by a stunning shift in the Ohio electorate toward Republicans, a new Dispatch/Ohio Newspaper Poll shows. The survey reveals that Republicans now own a 10-point margin over Democrats among likely voters declaring a party allegiance. Just in September, Democrats had an 11-point bulge with registered voters. (That's a fairly dramatic shift, and it'd be nice to see the real numbers, rather than just the differences.** Whether it "fueled" the result isn't something the poll measured. It's something the reporter made up.)

Many national polls contain similar rapid swings as independent voters abandon the Democratic Party -- reversing a Democratic buildup displayed in the 2006 and 2008 elections. (How many national polls, how similar and how rapid? Sorry, I have trouble believing this assertion. The party-ID question in the poll I follow most regularly -- Opinion Dynamics, polling for Fox -- has changed significantly over the past 10 months, but not to the degree the Dispatch is reporting. That doesn't mean a lot. Public opinion in Ohio is probably different from public opinion in the nation, and any poll could be an outlier; the whole point of reporting your confidence levels is to remind people that, generally, one out of every 20 results is an outlier. But if you're going to make assertions about "many polls," you really ought to provide some evidence.)

And like many of those surveys, the new poll of Ohioans shows that Republicans are far more engaged in this year's election. The percentage of the GOP "extremely interested" in the 2010 vote is almost double that of Democrats. Typical Democratic constituencies, such as young voters, urban dwellers and blacks, also are showing less interest in this year's campaigns. (Once again, you're giving an assertion about "many of those surveys" without any evidence. Which ones? National? How big? Conducted by whom? In what population? Asking what questions? In the field when? This is an interesting result -- far more important, if you want my opinion, than how people say they'd vote in an election 11 months away, and probably a lot more relevant to the Massachusetts result as well. But I'm left to take it on faith, and nothing the paper has shown so far inspires much in the way of faith in its survey reporting.)

We should emphasize, as usual, that this isn't a bad poll. On the evidence, it's professionally conducted, valid, reliable, thrifty, brave, cheerful, reverent -- everything a poll ought to be. It's a relevant part of a big mosaic of public opinion. It's interesting, but it isn't very dramatic. And there's our other big lesson for the evening. The things that make polls valuable rarely make them worthy of the top spot on the front page. And the things that make them lede-worthy are usually not the things that make them worth taking note of.

There are far, far worse things you can do with assertions about public opinion -- this deck from the lede hed in today's Freep, for example. We can't say it's false; as far as we know, it might be perfectly true. Our problem is that we have no idea. "Anger at banks grows" isn't the sort of conclusion you can draw from talking to one guy who wishes he had gotten a loan for a new truck. The nicest thing you can say about this hed is that it's a fabrication. It's one I'm instinctively inclined to agree with, but that's not a valid excuse for making stuff up for the front page.

More about this later. And, of course, a large welcome to all you BBC fans who followed the space-time rift here yesterday and gave us what may be our one-day record for hits. Bloody Torchwood.

* There's a lovely anecdote in "The Boys on the Bus," if that hasn't made your reading list yet.
** Should you be even more cautious because we're comparing "registered voters" to "likely voters"? You're starting to get the idea.