Friday, August 29, 2014

Don't do any of this

If you have to lead the front page with a poll (and you don't), you should at least try to avoid contradicting your main hed in your deck -- or your lead graf in your second graf:

LANSING — Democratic challenger Mark Schauer has edged ahead of Gov. Rick Snyder by two percentage points, according to a new poll of likely voters in the Nov. 4 election.

The results from EPIC-MRA of Lansing come just before the Labor Day holiday which is seen as the kickoff of the main campaign season and were released exclusively to the Free Press, WXYZ-TV (Channel 7) and statewide media partners. A 2-point lead is within the poll’s margin of error of plus or minus 4 percentage points, meaning the race remains a statistical dead heat.


You can be leading, or you can be in a dead heat, but you can't be doing both at the same time. (No, there's no such thing as a "statistical dead heat," and if there was, it wouldn't be defined by a meaningless term like "within the poll's margin of error.") A safe conclusion from these results is that both candidates' support has changed since July. Schauer, the Democrat, is doing a little better, while Snyder, the Republican, is doing a little worse. Both changes (43% to 45% and 46% to 43%) are also "within the margin of error," but they're both also meaningful. The smaller one has roughly two chances in three of accurately reflecting the value in the whole population, rather than being an accident of sampling. A good headline summary might be "Race still hypothetically tight, and things probably changed a little, but still maybe not" which in turn is a good argument for not leading the paper with poll stories.

More stuff you shouldn't do: Confuse subgroup results with the whole sample. It's interesting to note that Snyder leads in Oakland County, as the last graf tells us, but if we don't know how much of the sample came from Oakland, it's next to useless. A subsample there of 60 (10% of the total) would have a margin of error, at 95% confidence, of nearly 13 percentage points, and if you like those odds, you can go down to the old state fairgrounds and stare wistfully at the site of the former stove. Give us at least a suggestion of the size.

And -- OK, maybe this one is "especially" -- you shouldn't let sources lie with impunity:

The Snyder campaign criticized the poll as an outlier.

“One out of every 10 polls is garbage and this is certainly an example,” said Snyder spokeswoman Emily Benavides.

“Every poll in this race has shown the governor with an average lead of four points and the most recent one showed him with a nine-point advantage because Michigan is coming back and Gov. Snyder is leading the way.”


Oh, where to start? No, there is no individual poll in this race showing the governor (that'll be Snyder) with an "average lead of four points," because polls don't do that. His lead in the other seven most recent polls ranges from 1 to 4 points. There is no meaningful "average" of polls with different sample sizes drawn from different populations. This sample is no more likely to be an outlier than the one in July showing her boss in the lead. (The standard 95% confidence level, of course, means that one out of every 20 polls -- not every 10 -- is likely to not reflect the population value within 1.96 times the standard error of proportion, but that's a really unquotable way of saying "garbage.") The poll showing a 9-point lead for Snyder not only isn't the latest, it isn't even the latest by this company.

The spokeswoman is simply lying, and a story that doesn't call her on it is a bad story. The comments from the other camp are no more substantive (which is a good argument for scrapping the tradition of asking campaigns to comment on statistical evidence), but at least they don't play fast and loose with available evidence. "My opponent is an enemy of freedom" is an opinion; "that poll is an outlier" is a fact claim. It should be held to a higher standard.

Labels: , ,

0 Comments:

Post a Comment

<< Home