OK science, awful journalism

Cancer cured! Mideast at peace! Plenty of room for some cheery social science reporting in that most traditional of hard-news spots, the upper right corner of 1A. The tug of guilt on Catholic teens is weakening! (Come on, Virginia.) Stop the presses!

Well, maybe not. Today's story is very, very slightly about some mildly interesting survey data and much more about "how the press expresses, validates and amplifies popular prejudices," as our friendly critics over to the Log put it. It might look like a story about changing mores and religious identification among teenagers, but it isn't; it's about how journalism internalizes cultural stereotypes and how those stereotypes become pivotal in what the news audience is told about "science." The initial fault is with the reporter:
This just in from the myth-busting department: Roman Catholic teens feel no more guilty than other U.S. teenagers.

... but the copydesk deserves a share of blame for hyping the evidence-free parts of the story into a hed that looks to be pretty much a complete fabrication:

Catholic teens feel guilt's tug weaken

Study finds shame losing its traction

Sound harsh? Well, a little. It's hard to say, because the story itself commits a prime offense: It talks about conclusions but doesn't give any results* -- at least none that bear on its premises. Those premises are why it's at the top of the front, so let's have a look at them:

If they cheated on an exam, lied to their parents or engaged in serious petting, it's not bearing down on their conscience, according to a study by UNC-Chapel Hill researchers. At least it's not making them feel more guilty than their non-Catholic peers.

The emotional fallout of transgressing the Catholic Church's long list of sins -- venial and mortal -- may be a thing of the past. Blame the decline of ruler-wielding nuns at Catholic schools, or assimilation into the wider society.

We'll get back to the cheating, lying and sex in a minute. The bigger question is "may be a thing of the past," which suggests that "Catholic guilt" is more than just the reporter's invention. And for that, we need the sort of stuff the story** doesn't bother to tell us. (Why any such outcome should be cause for "blame" is a different matter; that's a really jarring insertion of opinion in something that's trying to pass for straight news.)

The study, to be published this month in the Review of Religious Research, is based on data from the National Study of Youth and Religion conducted by sociologist Christian Smith, now at the University of Notre Dame and Stephen Vaisey, at UNC-CH. The survey included 3,290 teens, of whom 819 were Catholic -- about 24 percent, roughly equivalent to the proportion of Catholics in the U.S. population.

Ahem. This is the Intertubes Age, and it's pretty careless not to provide at least a few links to the sources, if you can't get to the study itself. Still, there's enough in that graf to start tracking down some details, and those details suggest that the premise in the third graf (thus, the hed in the prime news position on the front) is false. You can't speculate about some notional decline in "emotional fallout" because this isn't the sort of study that measures those things.

What kind of study is it? From the NSYR Web site, the project is a national panel survey of teens and religion that first went in the field in 2002. The story (n = 3,290) suggests that the numbers come from the first "wave" of the survey, when the respondents were all 13 to 17. The advantage of a panel survey is that it goes back and interviews the same people. This one's now in its third wave (and the respondents are 18 to 23), and when that's complete, the data will allow for some conclusions you can't get from a simple cross-section. You could plausibly ask whether there's a change in guilt among young Catholics or Protestants (or unbelievers; hold that thought) as they age, but not whether there's a change among Catholic teens, because you're no longer surveying the 13-17 bracket. And, of course, data starting in 2002 have nothing to do with the Sister Mary Elephant myth the reporter is invoking.

The survey asked teens 13 to 17: "In the last year, how often, if ever, have you found yourself feeling guilty about things in your life?" and "How much, if any, of those feelings of guilt do you think were caused by religious influences?"

Teens who went to confession, now called the sacrament of reconciliation, were no more likely to feel guilty than non-Catholic teens.

OK. I'm not sure why the writer is breaking out teens who went to confession*** (or whether that's something done in the forthcoming study), since the theme is "Catholic guilt," not "guilt related to practice." But if you want, here's a handy site that lets you create your own crosstabs from any two variables**** in the first-wave survey. And it's pretty clear that there's no significant difference between Catholic teens and "another kind of Christian" teens.

Fine. But also nowhere near addressing the hed, "Catholic teens feel guilt's tug weaken." That's an invention of the newspaper. You can't say the study found it, because the study didn't measure it. There's no "before" to compare the 2002 data to. (The deck compounds the offense by claiming that the study found something similar about "shame," which isn't even in the questionnaire. Nor are cheating, lying and sex play measured as elements of "guilt.")

The trouble with inventing results to fit your preconceptions is that when you start asking "experts" to evaluate the results you describe, you get what's known as "fruit of the poison tree":

Changing values probably also account for a drop in Catholic guilt, said the Rev. Joseph Vetter, Duke University Catholic chaplain. Catholics used to feel guilty for not attending Mass on the Sabbath, or for living together before marriage. For many, those behaviors are now common.

He's a chaplain, not a methodologist. When a reporter asks for his response to a "study" that finds "Catholic guilt" declining, his job isn't to tear the data apart; it's to answer what he assumes to be an honest question. Thus his opinion is meaningless. We have no idea if there's any "drop in Catholic guilt" for him to speculate about. You could also ask why those social changes affect Catholics and not others, but ... well, we don't know whether anybody else's ambient guilt level has changed either, do we?

Again, this story isn't about "bad science." It's about a small part of a precise but not especially earthshaking study whose most interesting aspects are several years down the road (and probably won't draw a second glance from the N&O). The problem is bad journalism: yanking a fact out of its context and jamming it into somebody's favorite cultural stereotype. Too bad a nice chunk of 1A real estate went to waste.

* Put a little less stereotypically, the finding -- some six-year-old data show no significant differences on teen self-reports of how often they feel guilty -- it seems a little less likely to make the front.

** Reporter and editors probably share the guilt; it must have been nice to pitch a 9-graf hold-to-the-front science tale with a happy ending.

*** There doesn't appear to be much difference between those who have done confession in the past year and those who haven't. There could be a significant difference on feeling guilt "occasionally" vs. "rarely", but that's a pretty flimsy place to start building a story.

**** Heh heh heh.