Inept survey story of the (still-young) year

Behold! A hack social science story that makes almost every mistake in the book, yet sees the light of day because ... well, why did it see the light of day?

Usually, the response is along the lines of "Oh, lighten up. It's a cute little story that says something about Our Times, so who cares if it has all that statisticky stuff in it?" The short answer is "you do," and it's short for something like this:
1) It's a poll, so there are specific limits on what sort of knowledge it can produce.
2) When you write a story that ignores those limits, you are doing what we call "making stuff up."
3) Made-up stuff can't say something about Our Times. But it says a lot about the skills and intentions of the people who write and publish it.

All right, let's have a look. It's barely 11 a.m. on Sunday, so don't go get a drink first:

American Protestants are more loyal to their toothpaste or toilet paper than to their religious denomination, making consumers more choosy about Charmin or Colgate than they are about church, a new survey shows.
(On second thought, get me one while you're up.) The participial phrase -- is it supposed to modify "Protestants," by the way? -- more or less repeats the independent clause, apparently so the writer can toss in some alliteration. But in addition to compounding the aesthetic offense, it reinforces the crimes against the data. We're ignoring three sources of error:
1) Sampling
2) Subgroup comparison
3) Question design

Experts say the findings may be more telling about Americans' views of the plethora of Protestant groups than how they choose between Quilted Northern and, say, Cottonelle.

OK, make it four. Featurizing isn't always an offense against the data, but it is here. The findings don't address whether "consumers" are more "choosy" about "Charmin" than "church"; they're about what a particular subset of Protestants might do. Same with "Americans' views" about the "plethora of Protestant groups." That's not what we're measuring, so the findings can't be "telling" about it.

If you hang on to the 12th graf (what's taking so long with the drinks out there?), you can find a few hints about the sample size: It's a "nationally representative online panel of 1,007 U.S. adults, including 471 respondents who attended a Christian congregation one or more times a month." Assuming our pollster got a genuine random sample, not merely a "representative" one, that'd yield a margin of sampling error of 4.5 points at 95 percent confidence (the pollster reports 4.4, not that the story bothers to tell you either way). Let's be generous and say 400 of those count as Protestants; now we're at 4.9 percentage points. Why is that a big deal?

Researchers found 16 percent of Protestants say they would consider only one denomination, while 22 percent would use only one brand of toothpaste and 19 percent would use just one brand of bathroom tissue.

In other words, our sample could reflect a population value of about 11 to 21 percent for Protestants on "consider only one denomination" and about 14 to 24 percent on the toilet paper issue. But that overlooks another, possibly bigger, problem: Not all Protestants are equal on this question. Evangelicals are at 19 percent (dead even with Charmin), mainliners at 14. So even if you ignore the likelihood that the finding came about by chance, it's fundamentally dishonest to extend the toilet paper claim to all Protestants.

Here's how the pollster got to the consider-other-denominations issue (again, it's the reporter's job to ask questions like this, and if the reporter didn't, the story needs to go on the spike until the questions are answered): "Then they were asked what role that denomination would play if they could no longer attend their current church (if the church closed or if they moved to another area, for instance)."

See the question-design problem? How was that phrased for Charmin and Colgate and Cottonelle and all the other brand names the reporter threw in? Is there even a comparable way of framing that choice? ("Would you rather let your soul rot, let your teeth rot, use the Land's End catalog, none of the above, or don't know?")

When the first expert you ask for some context comes up with a comment like this:
“When you actually think about it for more than 10 seconds,” he said, “none of this is all that surprising and I don't think it's actually bad.”

... you might want to take that as a hint that you're making a mountain out of a pile of -- um, Charmin and Colgate. (Especially the "think about it for more than 10 seconds" bit.) Of course, the expert isn't the first person quoted in the story. That honor goes to -- the president of the marketing research firm that ran the survey!

Do you sense a larger concern emerging here? The survey doesn't really say anything new or interesting about denominational choice; it throws some giggly factoids at each other by way of getting your attention. Ellison Research gets some free publicity, the writer gets to play with alliteration, and the readers get -- what about the poor readers anyway?
1) They might believe you, in which case you've misled them.
2) They might not believe you -- in other words, they might have noticed that you ran a dishonest story, and they might infer from that evidence that any story you run about polling data (or religion, for that matter) is similarly dishonest.

I don't see a lot of merit in either outcome. If you need stuff to fill the "faith and values" page, it's not too late to consider the Washburo's look at the state of play on efforts to teach creationism in science classes. That has the merit of being true (as long as you edit the second graf carefully), informative and alliteration-free.