Here's a potentially interesting sign of things to come: a major-party campaign going after a story it dislikes through a new-media third party. The implications of that are more interesting than a little small-scale statistical whingeing -- though we are going to have a touch of that (and just as a reminder, if your first response to "margin of error" isn't "at what confidence level?", you are
not in shape for the stretch run). And even though the New York Times is still unmatched at stumbling over its own self-importance, the overall outlook is mildly cheery.
OK. So over at Talking Points Memo, we find
a report that the Obama campaign is a little miffed at today's NYT -- specifically, it has produced a lengthy complaint about how the
Times reported its own most recent omnibus poll on race relations, the State of the Nation, the general election campaign and just about everything else since Krakatoa (don't be intimidated by the
length of the NYT's pdf; it actually includes a lot of useful longitudinal stuff about MIP and presidential-performance questions and the like).
There's nothing new about going to a friendly outfit to gripe about what the Big Bad Press did, but there are a few details that make the complaint worth a look. More to the point, there's also a reponse from the NYT reporter -- and while we're at it, let's not forget to give the Times credit for posting the complete results; if you're inclined, you can take a run at assessing their credibility on your own. As improvements in the general state of campaign discourse go, those are hardly inconsequential.
First, the complainant, so over to TPM:
The Obama campaign sent over a detailed critique of the story, which concludes from the poll that Obama isn't closing the divide on race. The story's lead reporter was the paper's top political writer, Adam Nagourney.
"The NYT story about their poll ignores multiple and significant pieces of data that actually indicate a trend much different from that which the story suggests," the critique reads. It goes on to list "some straightforward points from their data that are omitted from the story."...
OK. You can check the list out yourself, but the bulk of the Obama-side details come down to "you didn't write the story we would have." They're welcome to that opinion, but absent a showing that (a) the Times is wrong and (b) the alternate version is righter or better (or both), they aren't going to get anywhere. (Complaining that the Times didn't report a stat that meets a criterion that an "independent expert" said would assure a win for Obama is just silly.) Two points are worth noting, though:
"Racial dissension" around Mrs. Obama's 24% favorable rating among whites is an extremely odd description given that Mrs. McCain's favorable rating among white voters is 20%.Right target, wrong arrow. Here's what the Times said:
There was even racial dissension over Mr. Obama’s wife, Michelle: She was viewed favorably by 58 percent of black voters, compared with 24 percent of white voters.I'm not convinced that comparison represents "dissension." More likely is the proportion of "not favorable" responses by race: 19% among whites, 1% among blacks. The grain of salt you need to take all those numbers with is that barely a quarter of black voters (27%) "haven't heard enough" about Michelle Obama to form a judgment, compared with 38% of white voters. About 55% of voters say that about Cindy McCain (the racial subgroups are indistinguishable; I wish they'd broken gender out on this one). Judgment: The Times overreached, but the complaint missed the mark.
And this:
Though there is a six-point margin of error among black voters the NYT describes the 7-point change in black voters' views that whites had a better chance of getting ahead as slightly higher than 8 years ago. Given that the Times reports horserace questions as statistically even when the margin falls within the margin, it seems that this shift from seven years ago among black voters is well within the margin of error.Hmm. Even though the designated complainer doesn't quite know what he or she is talking about, it's nice to see someone throwing a flag on the Times's handling of survey data. First things first, "within the margin" doesn't designate a magic point at which results become cosmic. Second, seven points is not "within" a six-point margin. Third, the
poll in 2000 was a very different creature: it included 934 black respondents, compared with 297, and it's more distinctly focused on racial -- specifically, black-white -- issues. Summary? The Times would have done better just to note the difference, rather than characterizing it as "slightly higher" (and to emphasize the risk of comparing different surveys), but the complaint is poorly drawn and looks irrelevant.
Now let's pick on the Times a bit, because it has a few things to answer for. Some of them relate to statistical hygiene: Never report a margin of sampling error without a confidence level (you can find it in the pdf, but that's no excuse). Don't round the margin of sampling error to whole numbers. Don't give the N for the whole sample and the margin of error for a subgroup (like registered voters).
But for the real problem, let's look at the Times reporter's response to the Talking Points post, elided a bit here for emphasis:
We could have chosen to focus on any number of themes; we decided to focus mainly on what we could learn from the poll about how blacks, whites and Hispanics view politics and society at the critical moment. ... But we are comfortable that our story accurately captured the results on the questions that most struck us, those that sought to illuminate how blacks, whites and Hispanics see the United States at a moment when Senator Obama's candidacy is putting race front and center in a new way.
Well and good, but -- that's not the story you ran on the front today, is it?
Poll Finds Obama Isn’t Closing Divide on RaceAmericans are sharply divided by race heading into the first election in which an African-American will be a major-party presidential nominee, with blacks and whites holding vastly different views of Senator Barack Obama, the state of race relations and how black Americans are treated by society, according to the latest New York Times/CBS News poll.The results of the poll ... suggested that Mr. Obama’s candidacy, while generating high levels of enthusiasm among black voters, is not seen by them as evidence of significant improvement in race relations.Well, there's your problem. The poll doesn't "find" that, because the poll doesn't measure it. The way to find out whether people think Obama is closing a "divide on race" is -- ready for this? -- to ask them. The writer (oh, the shock) blames the hed:
The point of the story is that black respondents apparently do not see the fact of Mr. Obama's candidacy as evidence of significant improvement in race relations. The story does not suggest that there is some onus on Mr. Obama himself to be closing this divide. He has a slight point: The hed's active (Obama isn't doing something), and the corresponding passage in the story is passive (something isn't being seen by black voters). But changing the voice of the hed verb wouldn't fix the story, which would still rest on the same faulty assumptions.
Where did the Times go wrong? Look at the statistical conclusions that support the story. Blacks are significantly more likely than whites to say that race relations are bad and that there's been no progress in ending discrimination. Whites are significantly more likely to say that black complaints about discrimination are overblown. No surprises there -- but that's not the problem. Here's Question 64:
Have you ever felt you were stopped by the police just because of your race or ethnic background?About 40 percent of black respondents, unchanged from eight years ago, said yes. But how could that answer possibly be affected by Obama's candidacy? Whatever your taste in verb voice, what's your rationale for suggesting that this is some measure of whether some particular political event is or isn't closing some "divide"? Or, to take a broader point, whether the closure of that divide by this campaign is the sort of thing your poll can even address?
It's nice of the Times to make its methods public, and to answer its critics in a timely fashion, but openness (however encouraging) isn't a cure for cluelessness. If you're going to use survey data to talk about how a campaign might or might not have affected as broad a topic as "race relations," you need to start by asking relevant questions that can logically be related to the conclusions you want to address.
Lessons for editors? There's more to editing a poll story than back-checking the long division. When a story draws weighty answers from questions it doesn't even bother to ask, someone needs to call it in for questioning. Stupid conclusions don't magically become valid when they're blessed by the lead political writer from the Times.
Labels: numbers, NYT, polls