Polling sins: Lying is so rarely a good idea

Today's quiz: Is the president's approval rating higher or lower than it was last month? What do you say there, Fair 'n' Balanced Network?

... Forty-four percent of voters approve and 51 percent disapprove of President Obama’s job performance, according to a Fox News poll released Friday.

Last month, 42 percent approved and 48 percent disapproved.

Judging from the headline, then, his approval rating fell from 42 percent to 44 percent. Aren't you just on the edge of your seat waiting for the next Fox report on gasoline prices?

We can, and should, raise a lot of questions about what this 2-point change means, if it means anything at all. How does this poll compare with others like it this week? Is any change from last month's version of the same poll conclusive, or is there too much wash in there to judge?

Before we do those, though, let's take a moment to point out what's right about Fox polling. Whatever your friends or your stylebook might say, it really doesn't matter whether a poll is funded by the forces of light or the forces of darkness. Methods are what counts: What questions did you ask? How did you draw the sample? How did you proceed in the field?

On those standards, the number we're looking at -- the proportion answering "approve" or "disapprove" on the presidential job performance question -- is pretty solid. Fox changed pollsters in February; its new agency stays in the field a little longer and gets a (very) slightly bigger sample than the previous one, but on a straightforward question like approval, there's no reason to doubt the result.

What sort of change the result represents is a different matter. Fox reports the margin of sampling error as 3 percentage points for the whole sample, which is wrong (it's a little over 3.2), but it gets credit for giving the confidence intervals for subgroups too. The change from last month is nonsignificant at traditional confidence levels. Obama's approval rating might well have risen a little, but that change also might well reflect the normal sort of churn that occurs when you estimate from a sample to a population.

An increase of 42% to 44% in the sample value could also reflect an actual decline in the population value. Approval of Obama's job performance among all registered voters could have fallen from 42% to 41%, and a sample value of 44% would still be within one tick* of the real thing.

It's easy to visualize this if you draw a horizontal line for the percentages, with vertical lines at each month's value.* From each vertical line, draw another horizontal line to represent the "margin of error" -- for Obama's December result, you'd start with 44 and draw a line from 40.8 (44-3.2) to 47.2 (44+3.2). Then pretend each vertical line is the center of a little Christmas bell!*** The more your bells overlap, the more cautious you want to be in saying that 44 is conclusively bigger than 42. (Refer back to the principles of survey reporting: Polls are rarely as interesting as you think they are.)

That abundance of caution should explain why we're wary about saying that a 2-point increase is a real increase, but it should also indicate why a 2-point increase is much less likely to be a "drop." Overenthusiasm in the first case makes one a statistical klutz; overenthusiasm in the second case is strongly correlated with lying.

And that has to be a problem if you're contracted to find data for our friends at the Fair 'n' Balanced Network. You can do a perfectly creditable job at all the things that make polling valid and reliable; on the evidence, that's what the professionals who run the Fox News Poll do. But when the information leaves your hands, it passes to people who tell lies as a matter of professional duty, career advancement, and sheer revulsion at the thought of having a Kenyan Muslim Communist defiling the White House. Must be a challenge explaining all that on the old CV.

* Standard error of proportion. Multiply it by 1.96 to get the "margin of error" at 95% confidence.
** For comparing two candidates in the same poll, draw a vertical line for each candidate. Same idea.
*** Ding dong merrily on high, and don't forget to report those effect sizes!