Man bites Times
Here's a potentially interesting sign of things to come: a major-party campaign going after a story it dislikes through a new-media third party. The implications of that are more interesting than a little small-scale statistical whingeing -- though we are going to have a touch of that (and just as a reminder, if your first response to "margin of error" isn't "at what confidence level?", you are not in shape for the stretch run). And even though the New York Times is still unmatched at stumbling over its own self-importance, the overall outlook is mildly cheery.
OK. So over at Talking Points Memo, we find a report that the Obama campaign is a little miffed at today's NYT -- specifically, it has produced a lengthy complaint about how the Times reported its own most recent omnibus poll on race relations, the State of the Nation, the general election campaign and just about everything else since Krakatoa (don't be intimidated by the length of the NYT's pdf; it actually includes a lot of useful longitudinal stuff about MIP and presidential-performance questions and the like).
There's nothing new about going to a friendly outfit to gripe about what the Big Bad Press did, but there are a few details that make the complaint worth a look. More to the point, there's also a reponse from the NYT reporter -- and while we're at it, let's not forget to give the Times credit for posting the complete results; if you're inclined, you can take a run at assessing their credibility on your own. As improvements in the general state of campaign discourse go, those are hardly inconsequential.
First, the complainant, so over to TPM:
The Obama campaign sent over a detailed critique of the story, which concludes from the poll that Obama isn't closing the divide on race. The story's lead reporter was the paper's top political writer, Adam Nagourney.
"The NYT story about their poll ignores multiple and significant pieces of data that actually indicate a trend much different from that which the story suggests," the critique reads. It goes on to list "some straightforward points from their data that are omitted from the story."...
OK. You can check the list out yourself, but the bulk of the Obama-side details come down to "you didn't write the story we would have." They're welcome to that opinion, but absent a showing that (a) the Times is wrong and (b) the alternate version is righter or better (or both), they aren't going to get anywhere. (Complaining that the Times didn't report a stat that meets a criterion that an "independent expert" said would assure a win for Obama is just silly.) Two points are worth noting, though:
"Racial dissension" around Mrs. Obama's 24% favorable rating among whites is an extremely odd description given that Mrs. McCain's favorable rating among white voters is 20%.
Right target, wrong arrow. Here's what the Times said:
There was even racial dissension over Mr. Obama’s wife, Michelle: She was viewed favorably by 58 percent of black voters, compared with 24 percent of white voters.
I'm not convinced that comparison represents "dissension." More likely is the proportion of "not favorable" responses by race: 19% among whites, 1% among blacks. The grain of salt you need to take all those numbers with is that barely a quarter of black voters (27%) "haven't heard enough" about Michelle Obama to form a judgment, compared with 38% of white voters. About 55% of voters say that about Cindy McCain (the racial subgroups are indistinguishable; I wish they'd broken gender out on this one). Judgment: The Times overreached, but the complaint missed the mark.
And this:
Though there is a six-point margin of error among black voters the NYT describes the 7-point change in black voters' views that whites had a better chance of getting ahead as slightly higher than 8 years ago. Given that the Times reports horserace questions as statistically even when the margin falls within the margin, it seems that this shift from seven years ago among black voters is well within the margin of error.
Hmm. Even though the designated complainer doesn't quite know what he or she is talking about, it's nice to see someone throwing a flag on the Times's handling of survey data. First things first, "within the margin" doesn't designate a magic point at which results become cosmic. Second, seven points is not "within" a six-point margin. Third, the poll in 2000 was a very different creature: it included 934 black respondents, compared with 297, and it's more distinctly focused on racial -- specifically, black-white -- issues. Summary? The Times would have done better just to note the difference, rather than characterizing it as "slightly higher" (and to emphasize the risk of comparing different surveys), but the complaint is poorly drawn and looks irrelevant.
Now let's pick on the Times a bit, because it has a few things to answer for. Some of them relate to statistical hygiene: Never report a margin of sampling error without a confidence level (you can find it in the pdf, but that's no excuse). Don't round the margin of sampling error to whole numbers. Don't give the N for the whole sample and the margin of error for a subgroup (like registered voters).
But for the real problem, let's look at the Times reporter's response to the Talking Points post, elided a bit here for emphasis:
We could have chosen to focus on any number of themes; we decided to focus mainly on what we could learn from the poll about how blacks, whites and Hispanics view politics and society at the critical moment. ... But we are comfortable that our story accurately captured the results on the questions that most struck us, those that sought to illuminate how blacks, whites and Hispanics see the United States at a moment when Senator Obama's candidacy is putting race front and center in a new way.
Well and good, but -- that's not the story you ran on the front today, is it?
Poll Finds Obama Isn’t Closing Divide on Race
Americans are sharply divided by race heading into the first election in which an African-American will be a major-party presidential nominee, with blacks and whites holding vastly different views of Senator Barack Obama, the state of race relations and how black Americans are treated by society, according to the latest New York Times/CBS News poll.
The results of the poll ... suggested that Mr. Obama’s candidacy, while generating high levels of enthusiasm among black voters, is not seen by them as evidence of significant improvement in race relations.
Well, there's your problem. The poll doesn't "find" that, because the poll doesn't measure it. The way to find out whether people think Obama is closing a "divide on race" is -- ready for this? -- to ask them. The writer (oh, the shock) blames the hed:
The point of the story is that black respondents apparently do not see the fact of Mr. Obama's candidacy as evidence of significant improvement in race relations. The story does not suggest that there is some onus on Mr. Obama himself to be closing this divide.
He has a slight point: The hed's active (Obama isn't doing something), and the corresponding passage in the story is passive (something isn't being seen by black voters). But changing the voice of the hed verb wouldn't fix the story, which would still rest on the same faulty assumptions.
Where did the Times go wrong? Look at the statistical conclusions that support the story. Blacks are significantly more likely than whites to say that race relations are bad and that there's been no progress in ending discrimination. Whites are significantly more likely to say that black complaints about discrimination are overblown. No surprises there -- but that's not the problem. Here's Question 64:
Have you ever felt you were stopped by the police just because of your race or ethnic background?
About 40 percent of black respondents, unchanged from eight years ago, said yes. But how could that answer possibly be affected by Obama's candidacy? Whatever your taste in verb voice, what's your rationale for suggesting that this is some measure of whether some particular political event is or isn't closing some "divide"? Or, to take a broader point, whether the closure of that divide by this campaign is the sort of thing your poll can even address?
It's nice of the Times to make its methods public, and to answer its critics in a timely fashion, but openness (however encouraging) isn't a cure for cluelessness. If you're going to use survey data to talk about how a campaign might or might not have affected as broad a topic as "race relations," you need to start by asking relevant questions that can logically be related to the conclusions you want to address.
Lessons for editors? There's more to editing a poll story than back-checking the long division. When a story draws weighty answers from questions it doesn't even bother to ask, someone needs to call it in for questioning. Stupid conclusions don't magically become valid when they're blessed by the lead political writer from the Times.
OK. So over at Talking Points Memo, we find a report that the Obama campaign is a little miffed at today's NYT -- specifically, it has produced a lengthy complaint about how the Times reported its own most recent omnibus poll on race relations, the State of the Nation, the general election campaign and just about everything else since Krakatoa (don't be intimidated by the length of the NYT's pdf; it actually includes a lot of useful longitudinal stuff about MIP and presidential-performance questions and the like).
There's nothing new about going to a friendly outfit to gripe about what the Big Bad Press did, but there are a few details that make the complaint worth a look. More to the point, there's also a reponse from the NYT reporter -- and while we're at it, let's not forget to give the Times credit for posting the complete results; if you're inclined, you can take a run at assessing their credibility on your own. As improvements in the general state of campaign discourse go, those are hardly inconsequential.
First, the complainant, so over to TPM:
The Obama campaign sent over a detailed critique of the story, which concludes from the poll that Obama isn't closing the divide on race. The story's lead reporter was the paper's top political writer, Adam Nagourney.
"The NYT story about their poll ignores multiple and significant pieces of data that actually indicate a trend much different from that which the story suggests," the critique reads. It goes on to list "some straightforward points from their data that are omitted from the story."...
OK. You can check the list out yourself, but the bulk of the Obama-side details come down to "you didn't write the story we would have." They're welcome to that opinion, but absent a showing that (a) the Times is wrong and (b) the alternate version is righter or better (or both), they aren't going to get anywhere. (Complaining that the Times didn't report a stat that meets a criterion that an "independent expert" said would assure a win for Obama is just silly.) Two points are worth noting, though:
"Racial dissension" around Mrs. Obama's 24% favorable rating among whites is an extremely odd description given that Mrs. McCain's favorable rating among white voters is 20%.
Right target, wrong arrow. Here's what the Times said:
There was even racial dissension over Mr. Obama’s wife, Michelle: She was viewed favorably by 58 percent of black voters, compared with 24 percent of white voters.
I'm not convinced that comparison represents "dissension." More likely is the proportion of "not favorable" responses by race: 19% among whites, 1% among blacks. The grain of salt you need to take all those numbers with is that barely a quarter of black voters (27%) "haven't heard enough" about Michelle Obama to form a judgment, compared with 38% of white voters. About 55% of voters say that about Cindy McCain (the racial subgroups are indistinguishable; I wish they'd broken gender out on this one). Judgment: The Times overreached, but the complaint missed the mark.
And this:
Though there is a six-point margin of error among black voters the NYT describes the 7-point change in black voters' views that whites had a better chance of getting ahead as slightly higher than 8 years ago. Given that the Times reports horserace questions as statistically even when the margin falls within the margin, it seems that this shift from seven years ago among black voters is well within the margin of error.
Hmm. Even though the designated complainer doesn't quite know what he or she is talking about, it's nice to see someone throwing a flag on the Times's handling of survey data. First things first, "within the margin" doesn't designate a magic point at which results become cosmic. Second, seven points is not "within" a six-point margin. Third, the poll in 2000 was a very different creature: it included 934 black respondents, compared with 297, and it's more distinctly focused on racial -- specifically, black-white -- issues. Summary? The Times would have done better just to note the difference, rather than characterizing it as "slightly higher" (and to emphasize the risk of comparing different surveys), but the complaint is poorly drawn and looks irrelevant.
Now let's pick on the Times a bit, because it has a few things to answer for. Some of them relate to statistical hygiene: Never report a margin of sampling error without a confidence level (you can find it in the pdf, but that's no excuse). Don't round the margin of sampling error to whole numbers. Don't give the N for the whole sample and the margin of error for a subgroup (like registered voters).
But for the real problem, let's look at the Times reporter's response to the Talking Points post, elided a bit here for emphasis:
We could have chosen to focus on any number of themes; we decided to focus mainly on what we could learn from the poll about how blacks, whites and Hispanics view politics and society at the critical moment. ... But we are comfortable that our story accurately captured the results on the questions that most struck us, those that sought to illuminate how blacks, whites and Hispanics see the United States at a moment when Senator Obama's candidacy is putting race front and center in a new way.
Well and good, but -- that's not the story you ran on the front today, is it?
Poll Finds Obama Isn’t Closing Divide on Race
Americans are sharply divided by race heading into the first election in which an African-American will be a major-party presidential nominee, with blacks and whites holding vastly different views of Senator Barack Obama, the state of race relations and how black Americans are treated by society, according to the latest New York Times/CBS News poll.
The results of the poll ... suggested that Mr. Obama’s candidacy, while generating high levels of enthusiasm among black voters, is not seen by them as evidence of significant improvement in race relations.
Well, there's your problem. The poll doesn't "find" that, because the poll doesn't measure it. The way to find out whether people think Obama is closing a "divide on race" is -- ready for this? -- to ask them. The writer (oh, the shock) blames the hed:
The point of the story is that black respondents apparently do not see the fact of Mr. Obama's candidacy as evidence of significant improvement in race relations. The story does not suggest that there is some onus on Mr. Obama himself to be closing this divide.
He has a slight point: The hed's active (Obama isn't doing something), and the corresponding passage in the story is passive (something isn't being seen by black voters). But changing the voice of the hed verb wouldn't fix the story, which would still rest on the same faulty assumptions.
Where did the Times go wrong? Look at the statistical conclusions that support the story. Blacks are significantly more likely than whites to say that race relations are bad and that there's been no progress in ending discrimination. Whites are significantly more likely to say that black complaints about discrimination are overblown. No surprises there -- but that's not the problem. Here's Question 64:
Have you ever felt you were stopped by the police just because of your race or ethnic background?
About 40 percent of black respondents, unchanged from eight years ago, said yes. But how could that answer possibly be affected by Obama's candidacy? Whatever your taste in verb voice, what's your rationale for suggesting that this is some measure of whether some particular political event is or isn't closing some "divide"? Or, to take a broader point, whether the closure of that divide by this campaign is the sort of thing your poll can even address?
It's nice of the Times to make its methods public, and to answer its critics in a timely fashion, but openness (however encouraging) isn't a cure for cluelessness. If you're going to use survey data to talk about how a campaign might or might not have affected as broad a topic as "race relations," you need to start by asking relevant questions that can logically be related to the conclusions you want to address.
Lessons for editors? There's more to editing a poll story than back-checking the long division. When a story draws weighty answers from questions it doesn't even bother to ask, someone needs to call it in for questioning. Stupid conclusions don't magically become valid when they're blessed by the lead political writer from the Times.
posted by fev at 7:46 PM
5 Comments:
What the poll respondents (and the poll) are looking at is simple: do Americans think that a black man's being a serious candidate for president shows that racism is dead?
That's all Obama has to do with it.
I don't know -- that's the question the story is looking at, but it's not what the poll looks at. That's where the Times falls flat. You can't measure what people think about the Obama candidacy's effect on racism by asking whether they've ever been pulled over by the cops because of race -- or by asking whether things have gotten better since the 1960s.
The poll has a lot of questions about Obama and a lot of questions about race, but I don't see anything that taps the impact of an Obama candidacy on racism. (It would have been a good question to ask, but not an easy one to write.)
It's not a bad poll. (In many ways, it's a really good one.) It's just not the poll the Times claims it to be, which makes the story a bad story.
I don't think it's asking about the impact of his candidacy on racism. I think it's asking if people think racism is dead because he's a candidate. That's not quite the same question. But I confess to being too busy this week to read the whole poll, or even (I admit) the story. So neither question may be the right one.
Maybe what I mean is, the poll is asking for people's opinions on racism, and then the reporter is saying - look, there's a black man running for president AND STILL people think such-and-such.
I don't think I'm expressing myself well ...
(well, in hindsight, almost anything's more articulate than the original post -- yeesh)
Where the Times goes wrong (and no, the Times wasn't camped out on my doorstep when I got home today waiting for this assessment) is in adding up a bunch of retrospective stuff and drawing inferences about the present from it. Whether Obama or Clinton or Kucinich is the nominee isn't going to affect your answer to "have you ever been pulled over by the cops because you're black?" or "have you ever been a victim of racial discrimination?" The stability of the answers suggests that the underlying phenomena haven't changed a lot, which is sad but not surprising
Even if Driving While Black magically stopped being a crime on the day of Obama's acceptance, it'd take a few years before a new cohort of black drivers came along to the point where the DWB question changes significantly. That question might shed light on Obama's impact on race relations in 10 years, but it won't in 2008.
I'm kind of harping on that one a lot because it's so distinctly retrospective, but some of the prospective questions are less illuminating than meets the eye. There's a lot of significant change in responses to "Do you think a black guy can be elected president?" -- but it's not a hypothetical question in July 2008 the way it was in July 2000.
All those questions are interesting and relevant, but they don't add up to the answers the Times claimed to have. On balance, it's cool that the Times is addressing substantive criticism almost in real time (though neither the criticism nor the response is especially well done). So I'm not dismayed. I just like to point out that there's more to "error" than "sampling error."
Post a Comment
<< Home