Lying with stats: It never goes away
Ack. Here it looked like a couple weeks' vacation from beating up on the abuse of survey data, and here come Fox and its acolytes looking for more fun.
Bottom line, if you see a story about a purported documentary called "Media Malpractice," credited to an alleged filmmaker named John Ziegler and incorporating a postelection survey by Zogby, kill it on sight. If readers call up to ask why you're suppressing it, guffaw heartily at them and point out that it's a fake story promulgated by hacks and liars.
This isn't a complaint about the poll itself (I have one off-the-cuff methodological complaint), which appears to represent what it represents. The problem is with how the results are contextualized, which would be obscene if it wasn't so amateurish. So let's have a look at what our crusading filmmaker wants them to say:
On November 4th, 2008 millions of Americans were shocked that a man of Barack Obama's limited experience, extreme liberal positions and radical political alliances could be elected President of the United States. For many of these Americans, the explanation was rather simple... the news media, completely enamored with Obama, simply refused to do their job.
On Election day twelve Obama voters were interviewed extensively right after they voted to learn how the news media impacted their knowledge of what occurred during the campaign. These voters were chosen for their apparent intelligence/verbal abilities and willingness to express their opinions to a large audience. The rather shocking video below seeks to provide some insight into which information broke through the news media clutter and which did not.
Or, as he put it to Hannity and Colmes:
No, here's what I'm saying, Alan. What I'm saying is that the media coverage of this campaign was so scandalous, so beyond bias, into the realm of media malpractice, which is why I'm doing a documentary with that title — and that's why we did this film, at HowObamaGotElected.com — that the reality is the media coverage was so horrendous that Obama voters had no idea for what they were voting. They had no idea about some of the basic issues of the campaign, many of which you and Sean talked an awful lot about.
To boil it down, the contention that Obama voters were clueless robots rests on two main assertions: They're ignorant when it comes to baseline information anyway (they don't even know who controls Congress), and thus, when the liberal media caved in and refused to report important fictional plants from the Fox axis, those critically important points didn't sink in.
Sounds kind of scary, especially if you haven't noticed that research has spent some decades finding out what sorts of things the electorate knows and doesn't know. The first of the documentary's points is that a survey found -- well, let's quote Sean Hannity: "Nearly 60 percent could not correctly say which party controls Congress. Now, that's frightening."
Some people scare easily. The National Election Survey folks have been asking a comparable, if simpler, question for some time now: "Do you happen to know which party has the most members in the House of Representatives in Washington before the elections (this/last) month?"
Zogby's results ("Before this past election,* which political party controlled both houses of Congress?") found 42.6 percent of Obama voters saying the Democrats, which our documentarian is going to count as the "correct" response -- hence, 57.4% (with a confidence of interval of 4.4 points) were wrong. That's more than the NES found in 2004 (46%) but quite a bit less than in 2002 (72%), 1986 (67%), or 1982 (68%). And if we count the 13% "neither" response as correct -- I have no objection, given that the 51-49 Senate majority relied on two independents -- our total is right down there with the bulk of responses.
Our hero scoffs when Colmes asks whether McCain voters would have similar responses, even offering to bet the poll's expenses (doubled if he loses) to see. I'd be a bit more careful with my money. Democrats and Republicans were about even in answering the NES question wrong in '04, '02 and '98, with the Democrats significantly higher in '00, '96, '94, '92 (indies and apoliticals are significantly higher in all those years). Interestingly, conservatives did worse than liberals in '04, better in '00.
Anyway, you can grow old in a hurry worrying about the cluelessness of American voters (as the post-Labor Day stretch began in 2004, John Kerry was around 75% in name recognition). The core of Hero Filmmaker's contention is that Crucial Facts about the evil Obama couldn't have sunk in anyway because the media ignored them (hence the "malpractice" part). So let's look at a few of those.**
Some of the Shock Outrage stats are down to the pretty reasonable observation that people do better identifying Beatles than they do Supreme Court justices (amazing; people pay attention to popular culture). Some of the rest are -- erm, let's say, a little on the selective side. Nor is it by any means clear what we're measuring with them.
Take, for example, "Which candidate currently has a pregnant teenage daughter?" (93.6% said "Palin.") What's that indexing? "Candidates who have teenage daughters" is one possibility. So is "messages introduced by the McCain campaign" (the datum about Palin's daughter didn't just fall from the clear blue sky, if you recall). On that count, you'd have to conclude that the media were very effective at transmitting GOP campaign messages.
Here's another: "Which candidate said their policies would likely bankrupt the coal industry?" The answer our hero is looking for is "Obama," though the correct answer is "none" -- as noted earlier, the "I'll bankrupt the industry" bit is a patent lie, fabricated at the last minute from an interview that had been public for 10 months. It's interesting to note that nearly 12% of respondents believed Obama had made the claim; that's really impressive for a lie that was introduced literally the day before the election (and ignored by many professional news organizations). On the other hand, 27% thought the claim was McCain's --- suggesting that when people guess at a WTF question, they really, really guess.***
The foamy-mouthed right should take solace in the results on "Which candidate said that Obama would be tested in his first six months as president by a generated international crisis?" -- most people said Biden, with only 21% saying McCain. If that strikes you as the sort of question that makes "inside baseball" sound user-friendly, you have a point. Seriously: On what planet, under how many suns, does it matter the square root of doodly-boo which candidate said that? What level of "issue knowledge" are you pretending that represents, and why?
We can leave out the questions about one-off disfluencies. When people venture a guess, most of them "know" that Obama had "claimed to have campaigned in 57 states." How that would compare with the proportion who might know that McCain "couldn't agree more" that Western Pennsylvania was a racist backwater? The honest answer is: Who cares? People slip when they're on stage 25 hours a day. Are you playing bloopers or politics?
But that does suggest a deeper concern. Respondents seem to have a pretty good idea about "which candidate said that the government should redistribute the wealth" -- certainly compared with "which candidate started their political career at the home of two former members of the Weather Underground?" (I can put up with singular "they," but "started their political career" entails the sort of multiple judgment that makes for really incompetent survey work.) But I'm missing a lot of the discourse that I found really interesting during the last few months of the campaign, which could have suggested questions like:
Which candidate thinks that nobody shot at anybody during the Cold War?****
Which candidate professes to love Israel but declared in a campaign speech that Israelis are not free?
Those get a little closer to "issue knowledge" (as opposed to Beatles-vs.-judges knowledge) -- or at least, in my view, to the difference between an occasional verbal fumble and a genuinely deep-seated indifference (bordering on stupidity) about how the world works. If your answer is "dunno," you can blame the Liberal Media that either edited Sarah Palin kindly, failed to connect the logical dots of her public babblings, or simply decreed (as did my local paper) that she had a firm grasp of the issues.
Enough of that. If you see a story based on the documentary, set it out on the midden for the wolves. And if John Ziegler waves his double-or-nothing around too often, he shouldn't be surprised if somebody takes it.
[UPDATED to fix the spelling of Ziegler's name, per his Web site (that'll teach your editor to rely on Fox's spelling. Tnx to Andy for pointing to the Nate Silver interview in the comments.]
* I wonder if "this past election" didn't cause some double-clutching. I like the NES phrasing -- "the election this month" -- better. Can't be too straightforward with survey questions.
** Again, this isn't Zogby's fault. Somebody pays your price, there's no reason not to run the poll. And there's no reason I can see to doubt the poll's quantitative results.
*** Only 47% of respondents picked a candidate for this question. Make of that what you will.
**** Since she seems so impressed by aviators who were captured by the commies -- Gov. Palin, meet Francis Gary Powers.
Bottom line, if you see a story about a purported documentary called "Media Malpractice," credited to an alleged filmmaker named John Ziegler and incorporating a postelection survey by Zogby, kill it on sight. If readers call up to ask why you're suppressing it, guffaw heartily at them and point out that it's a fake story promulgated by hacks and liars.
This isn't a complaint about the poll itself (I have one off-the-cuff methodological complaint), which appears to represent what it represents. The problem is with how the results are contextualized, which would be obscene if it wasn't so amateurish. So let's have a look at what our crusading filmmaker wants them to say:
On November 4th, 2008 millions of Americans were shocked that a man of Barack Obama's limited experience, extreme liberal positions and radical political alliances could be elected President of the United States. For many of these Americans, the explanation was rather simple... the news media, completely enamored with Obama, simply refused to do their job.
On Election day twelve Obama voters were interviewed extensively right after they voted to learn how the news media impacted their knowledge of what occurred during the campaign. These voters were chosen for their apparent intelligence/verbal abilities and willingness to express their opinions to a large audience. The rather shocking video below seeks to provide some insight into which information broke through the news media clutter and which did not.
Or, as he put it to Hannity and Colmes:
No, here's what I'm saying, Alan. What I'm saying is that the media coverage of this campaign was so scandalous, so beyond bias, into the realm of media malpractice, which is why I'm doing a documentary with that title — and that's why we did this film, at HowObamaGotElected.com — that the reality is the media coverage was so horrendous that Obama voters had no idea for what they were voting. They had no idea about some of the basic issues of the campaign, many of which you and Sean talked an awful lot about.
To boil it down, the contention that Obama voters were clueless robots rests on two main assertions: They're ignorant when it comes to baseline information anyway (they don't even know who controls Congress), and thus, when the liberal media caved in and refused to report important fictional plants from the Fox axis, those critically important points didn't sink in.
Sounds kind of scary, especially if you haven't noticed that research has spent some decades finding out what sorts of things the electorate knows and doesn't know. The first of the documentary's points is that a survey found -- well, let's quote Sean Hannity: "Nearly 60 percent could not correctly say which party controls Congress. Now, that's frightening."
Some people scare easily. The National Election Survey folks have been asking a comparable, if simpler, question for some time now: "Do you happen to know which party has the most members in the House of Representatives in Washington before the elections (this/last) month?"
Zogby's results ("Before this past election,* which political party controlled both houses of Congress?") found 42.6 percent of Obama voters saying the Democrats, which our documentarian is going to count as the "correct" response -- hence, 57.4% (with a confidence of interval of 4.4 points) were wrong. That's more than the NES found in 2004 (46%) but quite a bit less than in 2002 (72%), 1986 (67%), or 1982 (68%). And if we count the 13% "neither" response as correct -- I have no objection, given that the 51-49 Senate majority relied on two independents -- our total is right down there with the bulk of responses.
Our hero scoffs when Colmes asks whether McCain voters would have similar responses, even offering to bet the poll's expenses (doubled if he loses) to see. I'd be a bit more careful with my money. Democrats and Republicans were about even in answering the NES question wrong in '04, '02 and '98, with the Democrats significantly higher in '00, '96, '94, '92 (indies and apoliticals are significantly higher in all those years). Interestingly, conservatives did worse than liberals in '04, better in '00.
Anyway, you can grow old in a hurry worrying about the cluelessness of American voters (as the post-Labor Day stretch began in 2004, John Kerry was around 75% in name recognition). The core of Hero Filmmaker's contention is that Crucial Facts about the evil Obama couldn't have sunk in anyway because the media ignored them (hence the "malpractice" part). So let's look at a few of those.**
Some of the Shock Outrage stats are down to the pretty reasonable observation that people do better identifying Beatles than they do Supreme Court justices (amazing; people pay attention to popular culture). Some of the rest are -- erm, let's say, a little on the selective side. Nor is it by any means clear what we're measuring with them.
Take, for example, "Which candidate currently has a pregnant teenage daughter?" (93.6% said "Palin.") What's that indexing? "Candidates who have teenage daughters" is one possibility. So is "messages introduced by the McCain campaign" (the datum about Palin's daughter didn't just fall from the clear blue sky, if you recall). On that count, you'd have to conclude that the media were very effective at transmitting GOP campaign messages.
Here's another: "Which candidate said their policies would likely bankrupt the coal industry?" The answer our hero is looking for is "Obama," though the correct answer is "none" -- as noted earlier, the "I'll bankrupt the industry" bit is a patent lie, fabricated at the last minute from an interview that had been public for 10 months. It's interesting to note that nearly 12% of respondents believed Obama had made the claim; that's really impressive for a lie that was introduced literally the day before the election (and ignored by many professional news organizations). On the other hand, 27% thought the claim was McCain's --- suggesting that when people guess at a WTF question, they really, really guess.***
The foamy-mouthed right should take solace in the results on "Which candidate said that Obama would be tested in his first six months as president by a generated international crisis?" -- most people said Biden, with only 21% saying McCain. If that strikes you as the sort of question that makes "inside baseball" sound user-friendly, you have a point. Seriously: On what planet, under how many suns, does it matter the square root of doodly-boo which candidate said that? What level of "issue knowledge" are you pretending that represents, and why?
We can leave out the questions about one-off disfluencies. When people venture a guess, most of them "know" that Obama had "claimed to have campaigned in 57 states." How that would compare with the proportion who might know that McCain "couldn't agree more" that Western Pennsylvania was a racist backwater? The honest answer is: Who cares? People slip when they're on stage 25 hours a day. Are you playing bloopers or politics?
But that does suggest a deeper concern. Respondents seem to have a pretty good idea about "which candidate said that the government should redistribute the wealth" -- certainly compared with "which candidate started their political career at the home of two former members of the Weather Underground?" (I can put up with singular "they," but "started their political career" entails the sort of multiple judgment that makes for really incompetent survey work.) But I'm missing a lot of the discourse that I found really interesting during the last few months of the campaign, which could have suggested questions like:
Which candidate thinks that nobody shot at anybody during the Cold War?****
Which candidate professes to love Israel but declared in a campaign speech that Israelis are not free?
Those get a little closer to "issue knowledge" (as opposed to Beatles-vs.-judges knowledge) -- or at least, in my view, to the difference between an occasional verbal fumble and a genuinely deep-seated indifference (bordering on stupidity) about how the world works. If your answer is "dunno," you can blame the Liberal Media that either edited Sarah Palin kindly, failed to connect the logical dots of her public babblings, or simply decreed (as did my local paper) that she had a firm grasp of the issues.
Enough of that. If you see a story based on the documentary, set it out on the midden for the wolves. And if John Ziegler waves his double-or-nothing around too often, he shouldn't be surprised if somebody takes it.
[UPDATED to fix the spelling of Ziegler's name, per his Web site (that'll teach your editor to rely on Fox's spelling. Tnx to Andy for pointing to the Nate Silver interview in the comments.]
* I wonder if "this past election" didn't cause some double-clutching. I like the NES phrasing -- "the election this month" -- better. Can't be too straightforward with survey questions.
** Again, this isn't Zogby's fault. Somebody pays your price, there's no reason not to run the poll. And there's no reason I can see to doubt the poll's quantitative results.
*** Only 47% of respondents picked a candidate for this question. Make of that what you will.
**** Since she seems so impressed by aviators who were captured by the commies -- Gov. Palin, meet Francis Gary Powers.
posted by fev at 9:27 PM
3 Comments:
Here's more:
http://tinyurl.com/6em2jc
Where is the news in this piece in the first place? Someone selects, on a basis declared to be subjective, 12 people and asks them some questions. It's not a poll; it's not a survey of policy-makers; it's not much of a man-in-the-street even; what the heck is this thing and why is it getting any airtime at all?
On the bright side (so far, at least), this little gem hasn't taken hold as "news," outside the usual fairly predictable places. One of the problems is that newsrooms don't usually have people who can look critically at the value of stuff that's presented as 'research', so an incoherent or dishonest survey is likely to get lumped in with the general run of 'poll says' stuff. (It's easier to be 'objective' by offering a balance of comments, rather than by assessing the quality of evidence.)
Fox, of course, isn't motivated to ask methodological questions about stuff it agrees with. Real news organizations ought to be, and so far, the wall doesn't seem to be cracking on this one. Nonetheless, keeping the fingers crossed is in order.
Post a Comment
<< Home