Significance vs. substance

One last look at the end-of-the-year nonsense before we all shake the confetti out of the collective hair and get back to saving the world through copyediting. And what better way to keep up those resolutions about survey research than to discuss significance and substance?

Significance, you'll recall, doesn't always mean "importance." For today's purposes, it's a term of art referring to the likelihood -- based on an arbitrary level of confidence that you determine in advance -- that the difference you observe reflects a real difference in the population. In the AP story you see at right, for example, we can be confident that significantly more Americans expect scientists to find a cure for cancer in 2007 (35%) than expect scientists to find evidence of extraterrestrial life (19%). The miracle of random sampling assures us that there's less than a 1% chance that among all Americans, the alien faithful are tied with or leading the cure believers!

Substance, on the other hand, is the property that makes us ask (or should have made us ask before we put the AP story on the Monday front page, ahem): Who gives the southern end of a northbound rat how many Americans think we'll find extraterrestrial life this year? Or, to keep at least one foot on this planet, what proportion of Americans think there will be a terrorist attack, or an increase in gasoline prices, or anything else over which they have no influence?

If you have a particularly obsessive paper, or one that really doesn't look at wire content before it slaps the stuff online, you might have noticed this at the end: AP Manager of News Surveys Trevor Tompson, News Survey Specialist Dennis Junius and writer Kasie Hunt contributed to this report.

Truly the final indignity. We need the manager of news surveys and a survey specialist to sign off on a study that says more people expect a cure for cancer than expect proof of extraterrestrial life?

Speaking of which, the "survey" in the No. 3 spot on the Fox page has the other sort of problem. It must be fun to proclaim that "over 1/2 of married U.S. women not sure they'd marry husband again, over 1/3 say they definitely wouldn't, Web survey says," but it sure makes your news organization look -- oh, stupid and gullible? You can't tell whether this finding is "significant," of course, because nonrandom surveys (that'll include but not be limited to self-seleting online still) don't generate that sort of data. They're basically horoscopes on the newspage.

And speaking of horoscopes and extraterrestrial life: Dear friends at the Virginian-Pilot, whatever it was you were smoking when you fronted Pat Robertson's predictions (right), did you bring enough to share with the rest of the class?

OK. Here's the rule. Pat can talk to whoever he wants. But if he's going to proclaim it's God, you need to either confirm it with the other party or -- in the immortal words of Walter Burns -- stick him in the funny pages.

VIRGINIA BEACH -- Religious broadcaster Pat Robertson cited communications with God in predicting Tuesday that horrific terrorism aimed at the United States will result in "mass killing" during the second half of 2007.

"The Lord didn't say nuclear, but I do believe it'll be something like that -- that'll be a mass killing, possibly millions of people, major cities injured," Robertson said.

How was it He got that message across again? Semaphore? Igpay Atinlay? The Rock 'n' Roll Heaven Band spelling out "nuclear" as if it were Script Ohio?* And who let Pat out of J306 with this idea that you let a source off the hook like that? (And does God reduce the consonant cluster in "nuclear"?)

There's significance. Then there's substance. Then there's this:

Robertson said the actions of Israel's government also weighed heavily on God's heart and his.

"The word was that the Olmert policies were toxic" for Israel, said Robertson, referring to Israeli Prime Minister Ehud Olmert.

Robertson, who did not cite specific policies, said the United States was "feigning friendship" toward Israel while espousing policies that were "pushing them toward national suicide."

... His accuracy in predictions has been mixed, though Robertson on Tuesday claimed a "relatively good track record."

Please. Stop it.


* Bill Monroe is the dot. You heard it here first.