'Caught up in the moment'

It would have been nice to get through the summer without a polling rant, but -- OK, here's Nate Silver, stating the obvious:

We’ve documented for years how polls tend to rise and fall — in what are often fairly predictable patterns — after events like debates and conventions. In general, what suddenly goes up in polls tends to gradually come back down after a matter of a few weeks. Conventions typically produce polling swings of 4 to 6 percentage points toward the party that just nominated its candidate, for instance — but the polls usually revert back to about where they were before after a few weeks.

True, though surveywise it's not nearly as interesting as the "rally 'round the flag" effect. But stay tuned; conventional wisdom usually travels in flocks:

It looks as though something like this is happening again following the first Democratic debate last month. If you look at the RealClearPolitics average:

• Biden has rebounded to 28.4 percentage points from a low of 26.0 percentage points just after the debate. He was at 32.1 percent before the debate, so he’s regained about two-fifths of what he lost.
•  Harris has fallen to 12.2 percentage points from a peak of 15.2 percentage points. She was at 7.0 percent before the debate, so she’s lost about a third of what she’d gained.

Harris is still in better shape than she was before the debates, but she’s currently 16 points behind Biden instead of looking like she’s on the verge of overtaking him.

Which is the setup for this:

I’ll be honest … as predictable as this pattern is, it’s easy even for professionals like me to get caught up in the moment, especially in the early stages of a race before we’re using any sort of model to smooth the data out.

With (ahem) appropriate respect -- no, it isn't. That's one of the things that separate professionals from pundits. Medical professionals don't seem to have any trouble telling female patients not to waste time steaming their ladyparts with eucalyptus leaves, because part of the professional's job is distinguishing quackery from evidence. If you can't avoid getting "caught up in the moment," that's an argument against your claim to professionalism.

If a candidate rapidly goes from 7 to 15 in the polls, our unconscious, System 1 reflex is to assume the trend will continue, and that the candidate will continue gaining ground — to 20 points, 25 points and beyond. More often than not, though, the candidate loses ground after a sharp rise.

Unless we're claiming to be a professional psychologist or something, also no. We have no idea what "our" "unconscious" reflexive response to a perceived short-term rise in poorly aggregated survey data would be, because all we know about short-term responses to poorly aggregated survey data is what hucksters tell us. If professionals did their jobs, we wouldn't obsess about how people responded to headlines like zOMG RCP AVERAGE FALLS 3 POINTS ELECTION OVER!!11!11!1!1!!11!!!!, because they wouldn't occur in grownup publications. Yes, we'd still have to worry about the clueless and the openly fake, but that's why we have professionals: to spread the word as often as possible when bullshit needs to be countered.

This is a polling rant, but I'd like to think the point goes farther. The top-of-the-hour lead (most closely following a traffic report; make of that what you will) was about whether the "conventional wisdom" about Robert Mueller's congressional appearances this week had been borne out. Ladles and jellyspoons, those who want the conventional wisdom can find it anywhere; if you want to add some market value, could I suggest you start by providing unexcited news with frequent reminders that the null hypothesis can't be rejected based on the day's developments. 

Journalistic expertise is a fraught thing. Sad to say, one of the things we're best at is making data converge into story lines -- like the "game frame" that assumes, as above, that opening a lead in the first half points to opening a bigger lead in the second. That has the advantage of making stuff comprehensible (for many) and accessible (for many), and it's delivered a lot of actionable information about political, economic and cultural matters through the years. It also means we're traditionally not very good at telling stories that aren't interesting (like most survey results), even if they're likely to be more informative. We still have some time to work on that, and the week's events suggest that it would be nice to start sooner rather than later.

Oh, stop it

Is it time to complain about the misinterpretation of public opinion data again?  Yes, and Nate Silver is the one overplaying things here.

The point of the complaint seems to be that @MeetThePress is overemphasizing the stability of Trump's approval based on a single poll finding. To be fair, yeah: Most journalists writing about survey data would do well to dial back the loud adverbs, just on principle. But on the whole, there aren't a lot of reasons to reject the null here, and when in doubt, it's always good to err on the side of not jumping up and down for the sake of jumping up and down. Let's have a look at the data and see why.

The NBC/Journal poll (n = 600 registered voters, 44% approve, 52% disapprove) is the only one I know that was in the field entirely after the (ahem) events of Tuesday. The "margin of error" at 95% confidence is +/- 4 points. As Silver notes, the approval finding is down 2 points on an NBC poll that was in the field Aug. 18-22, and with 2 points being almost exactly the standard error,* he suggests that that's a bigger finding than "stable."

I think not, for a couple of reasons. NBC/WSJ surveys from June and July showed 44% and 45% approval, respectively, so one perfectly good interpretation of the approval result is "same as in June." More importantly, though, all samples are estimates. We can't prove that the population value (the actual figure that the samples are trying to estimate) has remained stable at 45% since June, but findings of 44-45-46-44 would be a really good representation of such a reality.

It's also entirely possible (when in doubt, bet the mean) that Trump's approval has inched up and then down since June at exactly the rate the samples indicate, or even a bit more or less. The findings could reflect a change in population value from 48% to 42%. They could also all reflect a steady population value of 48% or 42%; that would be a much less likely but still nonchance result (assuming identical sample sizes, which isn't spelled out in the pdf). And any of those results could be an outlier; a 95% confidence interval tells you that one of every 20 will be, but it doesn't tell you which one (or which 20). The safest conclusion about Trump's approval among registered voters since June is: Yep, pretty stable.

True, a decline of 2 points would be "reasonably big"** if "it held across other polls," and it'd be bigger if the SE was 1.5 instead of 2, but "other polls" haven't happened yet, and I'd rather that poll stories talk about what has been found rather than what might be. And "pretty stable" is what this poll has found -- like it or not.

That, I think, is the real issue: the urge to bash the methodology when you dislike the results of a survey. That's normal journalism, in that journalism finds it hard to resist pounding the data into the shape of a story, however hard the data resist. But business as usual -- specifically, the urge to proclaim that Trump's support was "tanking" when the data said no such thing -- runs the risk of making us look both biased and stupid. I'd prefer we aim for "neither."

* For approval, the square root of ((.44)(.56))/600, if you're scoring along at home; multiply by 1.96 to get the "margin of error" at 95% confidence. For simplicity, you can use .25 for the product, which would give the maximum margin for any finding based on the whole sample.
** See what I mean about the damn adverbs?