The case of the missing rally

It wouldn't be a "fake news" story if Fox wasn't somehow involved, would it?

President Trump is creeping up in the polls now that foreign policy has seized center stage.

...  It’s hardly surprising that Trump’s approval rating is getting a little better—up six points, to 40 percent, in a new Quinnipiac poll. He launched widely praised airstrikes against Syria over a chemical weapons attack, and used the Mother of All Bombs against ISIS in Afghanistan.

Fox, of course, is hardly the only news outlet to salivate over a phrase like "mother of all bombs." Nor is it the only one to link Trump's recent bellicosity to an improvement in public assessment of his performance; that would be a "rally 'round the flag" effect, of the sort that political scientists have been tracking for decades. What's interesting here is that Fox was only just catching up with the grownup media (at top is the Washington Post, claiming a "small boost in Trump's approval rating" in an April 14 article). And what's really interesting is that the evidence shows no rally effect. This isn't just fake news; it's fake news where there should be real news, and that's news.

Popular or not, the occupant of the White House tends to benefit when the world intrudes. As Matthew Baum put it, "presidents enjoy relatively short-lived spikes in their approval ratings immediately following sudden, high profile foreign policy events" (2002, p. 264). So I wasn't surprised when the Post's "Trump delights in watching the U.S, military display its strength" landed in the email feed:

The tough talk and the cruise missile strike have provided a small boost to Trump’s approval rating at home and have drawn widespread cable news coverage, distracting from his domestic struggles and the ongoing probe into contacts between his campaign officials and Russia. 

There are two claims here: that there's a non-incidental increase in Trump's approval rating, and that it was caused by "the tough talk and the cruise missile strike" (which happened April 6 our time). That's why copy editors don't get invited to parties where important people hang out: we're the ones who point out that unless the change is real and there are no confounding explanations for it, your claim of a causal relationship doesn't go in the paper. That's broadly the point of journalism: We try to tell you stuff that's true, and if we don't tell you something right away, we'd appreciate your patience while we try to figure out how true it is.

Here, a ratings bump is exactly what the rally effect would predict, but it isn't there. Gallup surveys immediately before and immediately after the missile attack saw a 1-point decline in Trump's approval; Rasmussen surveys in the field on the same dates found a 1-point increase. We fail to reject the null hypothesis.
But this flavor of fake news is like potato chips: Nobody eats just one. No one does a survey before proclaiming that the state is mourning, or the nation still reeling, or the town in shock; we just proclaim it, because that's the kind of context that ought to be true for news to make sense. In this case, theory agrees: Whatever grade of preening jackass occupies the White House, their approval ought to go up when they send a few missiles in the direction of the bad guys, because America.

That, I think, explains the Post's entirely unexamined assumption that Trump's approval somehow went up after he decided to attend to the world by waving his Trumphood at it. Then the New York Times decided to check in on the same topic:

Mr. Trump’s confrontational and improvisational approach to foreign affairs has lifted his mood, fortunes and poll numbers in recent days. There are signs it has also made an impact on the Chinese, prodding them to finally use their leverage with their errant neighbor, North Korea.

In a word, no. Even with several days' worth of data points in hand, there was no reason to believe either that Trump's "poll numbers" had risen or that any such increase could be laid to his "approach to foreign affairs." Let's look at some Trump approval data from three polls that were in the field at comparable times*: Gallup, sampling 1,500 adults at a go; Rasmussen, with 1,500 likely voters; and the Economist, sampling 1,271 to 1,331 registered voters. The April 9-11 period is the first one after the attack on Syria.

             3/19-21   3/26-28     4/2-4      4/9-11    4/15-19
Gallup     40%           35%            42%           41%           42%
Rasm       46%           44%            46%           47%          49%
Econ        44%           45%            43%           43%          44%

If you Marched for Science on Saturday, the first thing you'll probably note is that the chart doesn't "prove" anything. Exactly right. Trump's approval among US adults (Gallup) could have risen from 40% to 44% from the beginning to the middle of April; the 42% captured in those surveys could be a nonchance -- "within the margin of error," if you must -- representation of exactly such an increase. Equally, approval could have fallen by the same amount in that population -- or, among registered voters, wavered a tiny bit before returning to where it was in the middle of March, as the Economist suggests. The numbers don't prove anything, and one of the many things they don't prove is a rally effect.

What about the Rasmussen poll -- that's a bigger change than the "margin of error," isn't it? (Yes, unless the population value is somewhere between the two sample values.) Reuters, on the other hand, using a larger sample, found a similar decline in approval, 46% to 43%, after the Syria attack. As the feckless Howard Kurtz of Fox News notes above, the Quinnipiac poll did find a 5-point increase (35%-40%) among registered voters. But a 95% confidence level means that, on average, one sample of every 20 is an outlier; maybe Gallup's 35% finding (among the, ahem, 15 samples in the chart) and Quinnipiac's are both perfectly expectable underestimates of Trump's approval in their respective populations.

Wasn't that boring? Actually, I'd like to think not. The data don't "prove" that Trump didn't get a rally effect from whacking an airfield in Syria. (Allow me to quote one of my cherished research mentors: If you want proof, go to seminary.) But neither can anyone show that there was one, and that means Fox, and the Post, and the Times, are peddling fake news.

That's fake news, not false news; proclaiming the conventional wisdom that everyone else is proclaiming is one of the things journalists do, and it's not the same as making things up. The Post and the Times may have better intentions than Fox, and they're as likely to botch a story like this one in favor of candidates they oppose as in favor of candidates they support, but in this case the effect is the same. People are dumber -- or at least, equally dumb as before -- on a matter about which the press had a chance to make them fractionally smarter, and that's unfortunate.

If, on the other hand, the "rally effect" is actually broken, somebody should be looking into it.  That would be a pretty significant story.

* The Economist/YouGov poll was in the field April 10 and 11; Gallup and the Economist were in the field April 15-18, and Rasmussen April 17-19. Data and field dates taken from the right-wing linkblog RealClearPolitics, which usually reports results correctly even though its "average" is a known fake.
  
Baum, M.A. (2002). The constituent foundations of the rally-round-the-flag phenomenon. International Studies Quarterly, 46, 263-298.