National Lying with Statistics Day
Well, speak of the devil, and he will surely sneak over and put his beer bottles in with your recycling. Or worm his way into America's newsrooms and work his malevolent will on the desk:
FLORIDA GOVERNOR'S RACE
ARE THEY TIED?
Normally we'd ask something like "since you have the poll data, how about telling us?" But somebody's about to:
New poll gives Crist a negligible lead over Davis, bringing added importance to tonight's debate
Seems that would kind of render the Stupid Question moot, but for the annoying fact of the lede:
Florida's lopsided contest for governor has suddenly become a dead heat, according to a new poll that boosts the stakes for tonight's first televised debate between Republican Charlie Crist and Democrat Jim Davis.
The surprising results energized the longtime underdog, Davis, now trailing by only two percentage points in the Quinnipiac University survey, and put Crist on the defensive. Nearly every other poll has showed a double-digit gap.
The main centerpiece hed asks if they're tied, the deck says they aren't, the lede says that they are. Looks like the prisoners are toying with the guards a little down at the Herald. Or else the Evil One was beclouding minds across the nation -- his craft and power are great -- because Tuesday was a day of strikingly widespread offenses against the crafts of polling and journalism.
The worst and most consistent were in the outposts of the old K-R (now McClatchy) empire, emanating from the K-R side of the now-consolidated McClatchy Washburo. This suggests that someone needs to give that buro a good talking-to. If your claim to exclusivity and authority is that you can't be bothered to play by the rules, you're not doing your member papers (or those who, ahem, have your stock in their retirement portfolios) any favors.
Here's the offending lede and some of the heds it produced:
WASHINGTON - Republican Senate candidates have fought back to regain an edge in two key races, pivotal battlegrounds that could determine which party controls the Senate, according to a series of new McClatchy-MSNBC polls.
Charlotte:
Virginia, Tennessee called key to Senate control
Polls find GOP leads in both of the states; Dems need to win at least one
There's a particular editing sin in this version; the story (at least, as it appears on the Web) never says what the numbers are for these states. Good thing, too, since the polls don't actually find the GOP leading in either one. Slick, huh?
Wichita fell for the same snake oil:
GOP gains in two key races
And the Strib managed the cognitive dissonance centerpiece trick again (doncha love the dainty grammar of the lede-in?):
To which party will the Senate tilt? The answer depends on nine key races, with many ...
TOO CLOSE TO CALL
Exactly right! Go team! Too bad the big type with the photos highlights the percentages in the Virginia and Tennessee races, and the lede repeats the "fought back to regain an edge" assertion. Which, at bottom, amounts to Lying with Statistics, to steal somebody else's title.
Let's try some analogies, since sweet reason obviously hasn't worked so far. You can't play by half the rules. You can't put 11 guys on the field and say the other side is limited to five. If you're going to rely on the laws of probability, you can't throw them out when they get in the way of your thrilling, authoritative, edge-of-the-seat lede.
Now. Let's imagine a giant barrel -- we'll call it "Tennessee" -- holding 10,000 pingpong balls. Some of those pingpong balls are marked "Ford," some are marked "Corker" and some are marked "undecided" (we think most of them are lying -- they have too decided, but they aren't saying -- but polling is about what we know, not what we guess).
Here's how the K-R expert put it: Republican Bob Corker, the former mayor of Chattanooga, led Democrat Rep. Harold Ford Jr. by 45-43 percent. Three weeks ago, Ford led by 43-42 percent.
Let's do the same thing. We're going to pick some pingpong balls out at random every few weeks and see if we can guess the makeup of the whole population of "Tennessee." We know a couple of things:
1) The more pingpong balls we draw every week, the more likely we are to get closer to an accurate representation.
2) Random samples themselves will end up following a normal distribution. This is the magic part. It lets us draw some conclusions about where stuff lies in relation to other stuff. And once we decide how confident we want to be -- one chance in three, or 20, or 100, of being wrong -- and how many pingpong balls we'll pick, we know a lot about where other cases will lie on the normal distribution curve.
Based on the number of pingpong balls McClatchy picked (Charlotte cut the confidence level, which is bad editing), we know that "Ford 43" means Ford's support in "Tennessee" is probably somewhere between 39% and 47%. How probably? There's a 5% chance that "Ford 43" is actually "Ford 20" or "Ford 57" or something else outside that range. And "Corker 42" means that anything between "Corker 46" and "Corker 38" is likely at the same level.
So when one week's draw is Ford 43, Corker 42 and the next week's is Corker 45, Ford 43, what should we conclude? We ought to be pretty happy. We're making pretty good guesses about the population. Both candidates are pretty much somewhere between 40% and the mid- to upper 40s in support. No reason to stop the press for it, but it isn't uninteresting.
What we shouldn't conclude -- and to be blunt, which more McClatchy editors need to start being, we're either dishonest or stupid to conclude -- is that Satan snuck into the barrel and magically changed the ratio of pingpong balls. This survey has detected no change in the population. That's a pretty good finding. It's not a very exciting story, but it has the advantage of being true -- unlike the preposterous claim that Corker has clawed, or fought, or suplexed his way back into a race that was in essence tied the last time we looked.
Editors, I'm serious. You guys have to call an end to this stuff. Let me quote an old foreign-desk hand in a wake-up memo to overseas buros:
If you file shit and the AP files spun gold, you go on the spike. And the AP goes in the paper.
FLORIDA GOVERNOR'S RACE
ARE THEY TIED?
Normally we'd ask something like "since you have the poll data, how about telling us?" But somebody's about to:
New poll gives Crist a negligible lead over Davis, bringing added importance to tonight's debate
Seems that would kind of render the Stupid Question moot, but for the annoying fact of the lede:
Florida's lopsided contest for governor has suddenly become a dead heat, according to a new poll that boosts the stakes for tonight's first televised debate between Republican Charlie Crist and Democrat Jim Davis.
The surprising results energized the longtime underdog, Davis, now trailing by only two percentage points in the Quinnipiac University survey, and put Crist on the defensive. Nearly every other poll has showed a double-digit gap.
The main centerpiece hed asks if they're tied, the deck says they aren't, the lede says that they are. Looks like the prisoners are toying with the guards a little down at the Herald. Or else the Evil One was beclouding minds across the nation -- his craft and power are great -- because Tuesday was a day of strikingly widespread offenses against the crafts of polling and journalism.
The worst and most consistent were in the outposts of the old K-R (now McClatchy) empire, emanating from the K-R side of the now-consolidated McClatchy Washburo. This suggests that someone needs to give that buro a good talking-to. If your claim to exclusivity and authority is that you can't be bothered to play by the rules, you're not doing your member papers (or those who, ahem, have your stock in their retirement portfolios) any favors.
Here's the offending lede and some of the heds it produced:
WASHINGTON - Republican Senate candidates have fought back to regain an edge in two key races, pivotal battlegrounds that could determine which party controls the Senate, according to a series of new McClatchy-MSNBC polls.
Charlotte:
Virginia, Tennessee called key to Senate control
Polls find GOP leads in both of the states; Dems need to win at least one
There's a particular editing sin in this version; the story (at least, as it appears on the Web) never says what the numbers are for these states. Good thing, too, since the polls don't actually find the GOP leading in either one. Slick, huh?
Wichita fell for the same snake oil:
GOP gains in two key races
And the Strib managed the cognitive dissonance centerpiece trick again (doncha love the dainty grammar of the lede-in?):
To which party will the Senate tilt? The answer depends on nine key races, with many ...
TOO CLOSE TO CALL
Exactly right! Go team! Too bad the big type with the photos highlights the percentages in the Virginia and Tennessee races, and the lede repeats the "fought back to regain an edge" assertion. Which, at bottom, amounts to Lying with Statistics, to steal somebody else's title.
Let's try some analogies, since sweet reason obviously hasn't worked so far. You can't play by half the rules. You can't put 11 guys on the field and say the other side is limited to five. If you're going to rely on the laws of probability, you can't throw them out when they get in the way of your thrilling, authoritative, edge-of-the-seat lede.
Now. Let's imagine a giant barrel -- we'll call it "Tennessee" -- holding 10,000 pingpong balls. Some of those pingpong balls are marked "Ford," some are marked "Corker" and some are marked "undecided" (we think most of them are lying -- they have too decided, but they aren't saying -- but polling is about what we know, not what we guess).
Here's how the K-R expert put it: Republican Bob Corker, the former mayor of Chattanooga, led Democrat Rep. Harold Ford Jr. by 45-43 percent. Three weeks ago, Ford led by 43-42 percent.
Let's do the same thing. We're going to pick some pingpong balls out at random every few weeks and see if we can guess the makeup of the whole population of "Tennessee." We know a couple of things:
1) The more pingpong balls we draw every week, the more likely we are to get closer to an accurate representation.
2) Random samples themselves will end up following a normal distribution. This is the magic part. It lets us draw some conclusions about where stuff lies in relation to other stuff. And once we decide how confident we want to be -- one chance in three, or 20, or 100, of being wrong -- and how many pingpong balls we'll pick, we know a lot about where other cases will lie on the normal distribution curve.
Based on the number of pingpong balls McClatchy picked (Charlotte cut the confidence level, which is bad editing), we know that "Ford 43" means Ford's support in "Tennessee" is probably somewhere between 39% and 47%. How probably? There's a 5% chance that "Ford 43" is actually "Ford 20" or "Ford 57" or something else outside that range. And "Corker 42" means that anything between "Corker 46" and "Corker 38" is likely at the same level.
So when one week's draw is Ford 43, Corker 42 and the next week's is Corker 45, Ford 43, what should we conclude? We ought to be pretty happy. We're making pretty good guesses about the population. Both candidates are pretty much somewhere between 40% and the mid- to upper 40s in support. No reason to stop the press for it, but it isn't uninteresting.
What we shouldn't conclude -- and to be blunt, which more McClatchy editors need to start being, we're either dishonest or stupid to conclude -- is that Satan snuck into the barrel and magically changed the ratio of pingpong balls. This survey has detected no change in the population. That's a pretty good finding. It's not a very exciting story, but it has the advantage of being true -- unlike the preposterous claim that Corker has clawed, or fought, or suplexed his way back into a race that was in essence tied the last time we looked.
Editors, I'm serious. You guys have to call an end to this stuff. Let me quote an old foreign-desk hand in a wake-up memo to overseas buros:
If you file shit and the AP files spun gold, you go on the spike. And the AP goes in the paper.
posted by fev at 10:18 PM
6 Comments:
Perhaps we should hire a plane and carpet bomb the MCT bureau with old statistics texts. Not that it would do any good or that anyone would care -- but maybe it would knock them senseless until, say, Nov. 10, so we'd be spared from this crap.
Somehow I have this vision of Dale Dye calling in a logistic regression on his own position. It's worth a try. Know any daredevil piltos down your way?
I've been compiling all sorts of instructions and tips and ideas while I've been in this, my first editing job. When I find really good things, I like to share them with my desk. The others may already be fully aware of what I'm saying, but I wonder if my "youthful enthusiasm" might serve as a good reminder.
Since Election Day is next week and we'll have a desk meeting this week, may I share this post (as a hard-copy handout) with my desk? Of course, I will include as much attribution as possible -- I only know you as Fev, though. I'll include this blog address, at least.
Thank you for your blog. It has offered plenty of insight and been quite helpful for me.
~Beth (wordnerdy)
Hi, Beth: Welcome to the wonderful world of editing, tnx for the note, and feel free to share as you wish. Bibliographic &c details available at the address at right, headsuptheblog@cs.com.
K.C. Star ombudsman on polls:
http://www.kansascity.com/mld/kansascity/contact_us/about_np1/15874625.htm
Tnx a ton for the link to that fine column. Wish I wasn't so convinced in advance that it'll be completely ignored on the Star's news desk.
Two particularly good takeaway points:
1) The office politics of who paid for which poll in the Senate race are lost on most readers. Those who want to presume the relative play of stories reflects a bias have every reason to.
2) In most cases, polls _really aren't_ a very big story (Happy now, Strayhorn?). A finding that there's no significant shift in the population is, um, pretty dull stuff.
Long as I have the floor, two more pleas:
1) Let's ban the term "within the margin of error" from all future news copy. Good to know people are paying attention to statistics, but "within the margin" really isn't a relevant consideration. When you get beyond _twice_ the margin of sampling error, that's a big deal; it means you're 95% sure there's a real difference in the population. But with a 4-point margin, a 5-point difference in the sample isn't a whole lot more exciting than a 3-point difference. No matter what the stylebook says.
2) On first ref, pls make it "margin of sampling error," not "margin of error." There's a ton of error can happen in surveys. Sampling is only part of it.
Post a Comment
<< Home