'Caught up in the moment'

It would have been nice to get through the summer without a polling rant, but -- OK, here's Nate Silver, stating the obvious:

We’ve documented for years how polls tend to rise and fall — in what are often fairly predictable patterns — after events like debates and conventions. In general, what suddenly goes up in polls tends to gradually come back down after a matter of a few weeks. Conventions typically produce polling swings of 4 to 6 percentage points toward the party that just nominated its candidate, for instance — but the polls usually revert back to about where they were before after a few weeks.

True, though surveywise it's not nearly as interesting as the "rally 'round the flag" effect. But stay tuned; conventional wisdom usually travels in flocks:

It looks as though something like this is happening again following the first Democratic debate last month. If you look at the RealClearPolitics average:

• Biden has rebounded to 28.4 percentage points from a low of 26.0 percentage points just after the debate. He was at 32.1 percent before the debate, so he’s regained about two-fifths of what he lost.
•  Harris has fallen to 12.2 percentage points from a peak of 15.2 percentage points. She was at 7.0 percent before the debate, so she’s lost about a third of what she’d gained.

Harris is still in better shape than she was before the debates, but she’s currently 16 points behind Biden instead of looking like she’s on the verge of overtaking him.

Which is the setup for this:

I’ll be honest … as predictable as this pattern is, it’s easy even for professionals like me to get caught up in the moment, especially in the early stages of a race before we’re using any sort of model to smooth the data out.

With (ahem) appropriate respect -- no, it isn't. That's one of the things that separate professionals from pundits. Medical professionals don't seem to have any trouble telling female patients not to waste time steaming their ladyparts with eucalyptus leaves, because part of the professional's job is distinguishing quackery from evidence. If you can't avoid getting "caught up in the moment," that's an argument against your claim to professionalism.

If a candidate rapidly goes from 7 to 15 in the polls, our unconscious, System 1 reflex is to assume the trend will continue, and that the candidate will continue gaining ground — to 20 points, 25 points and beyond. More often than not, though, the candidate loses ground after a sharp rise.

Unless we're claiming to be a professional psychologist or something, also no. We have no idea what "our" "unconscious" reflexive response to a perceived short-term rise in poorly aggregated survey data would be, because all we know about short-term responses to poorly aggregated survey data is what hucksters tell us. If professionals did their jobs, we wouldn't obsess about how people responded to headlines like zOMG RCP AVERAGE FALLS 3 POINTS ELECTION OVER!!11!11!1!1!!11!!!!, because they wouldn't occur in grownup publications. Yes, we'd still have to worry about the clueless and the openly fake, but that's why we have professionals: to spread the word as often as possible when bullshit needs to be countered.

This is a polling rant, but I'd like to think the point goes farther. The top-of-the-hour lead (most closely following a traffic report; make of that what you will) was about whether the "conventional wisdom" about Robert Mueller's congressional appearances this week had been borne out. Ladles and jellyspoons, those who want the conventional wisdom can find it anywhere; if you want to add some market value, could I suggest you start by providing unexcited news with frequent reminders that the null hypothesis can't be rejected based on the day's developments. 

Journalistic expertise is a fraught thing. Sad to say, one of the things we're best at is making data converge into story lines -- like the "game frame" that assumes, as above, that opening a lead in the first half points to opening a bigger lead in the second. That has the advantage of making stuff comprehensible (for many) and accessible (for many), and it's delivered a lot of actionable information about political, economic and cultural matters through the years. It also means we're traditionally not very good at telling stories that aren't interesting (like most survey results), even if they're likely to be more informative. We still have some time to work on that, and the week's events suggest that it would be nice to start sooner rather than later.

Random babbling about innocent numbers

What's entertaining about the CNN polling analysis on Sunday morning's home page isn't whether it slips on a statistical banana peel or two, but how much of it bears any resemblance to things that actually happen in the political ecosystem. Shall we?

Poll of the week: A new NBC News/Wall Street Journal poll shows 50% of voters prefer November's election to result in a Democratic-controlled Congress compared with 42% of voters who prefer a Republican-controlled Congress.

I don't love the syntax; I'd go with "prefer that November's election result in ..." if some editor insisted that you can't say "want November's election to produce," but  it's not completely out of bounds for "what is your preference for the outcome of this November's congressional election."*

This result is in-line with other polls from NBC News/Wall Street Journal and from other generic ballot polls more generally. 

True so far, despite the irrelevant hyphen in "in-line." Since December in this poll, the result for Democrats has ranged from 47% to 50%; for Republicans, from 39% to 43%. "More generally" is sort of what "other generic ballot polls" means, but that's copy-editing, not reality vs. unreality. Which gets us to the paragraph reproduced above, which needs to be gone through more carefully.

At the most basic level, the generic ballot can be seen as an estimation of the House popular vote. 

Technically ... yeah, that's what the question tries to estimate. Whether it produces as good an estimation as "If the election for Congress were held today, would you vote for [ROTATE] the Democratic candidate in your district or the Republican candidate in your district?"** is another matter, but the sentence is misleading for a different reason. There's no Electoral College for the House; the "popular vote" is the only kind we have.

Given where the generic ballot average is at this point (an 8-point Democratic advantage), there is roughly an 6 to 8 point margin of error in predicting the final vote.

Wrong on multiple counts. By "generic ballot average," the writer may mean the RealClearPolitics average for generic ballot questions, which at this writing is given as 8.4 points. That "average," of course, is not a meaningful number; you could use a weighted mean to smooth out the eight different sample sizes it represents (RCP doesn't), but that doesn't cover the different populations sampled (registered voters vs. likely voters) or the different questions asked. "If the election were held today ..." and "what is your preference for ..." look like they're tapping the same sentiment, but still: You can't average the results of different questions. That's literally cheating.

The so-called "average" doesn't have a "margin of error," and RCP doesn't claim that it does, but if it did, the "margin" wouldn't be affected to that degree by the difference in proportions or by the time represented "at this point."*** Even if there was a meaningful "average," it wouldn't affect the "margin of error" as claimed. The difference in proportions is represented in the numerator of ((p)(1-p))/n, of which we take the square root to get a standard error of proportion. The 50% Democrat result in the NBC/WSJ sample produces a margin of error of 3.3 points at 95% confidence;**** the margin on the Republican result is 3.2. And the "margin of error" isn't related to the "final vote," which doesn't happen until 10 weeks after the survey concluded. We report sampling error because we can measure it, but we can't measure biases like nonreponse or social desirability. Fabricating a "margin" that apparently accounts for biases like those is just blowing smoke.

In other words, Democrats end up winning by double digits or roughly tied with Republicans. That's far from perfect, though is actually more predictive than an average for a Senate race would be at this point.  

If there was a "margin of error" involved here, that's just about exactly the opposite of how it would work. Assuming a 6-point lead and a 6-point margin, nonchance results could include either a double-digit win or a tie -- but those are the far ends of the distribution, and the bulk of findings from repeated samples would cluster around the actual population results and slope downward from there.

I don't know what is meant by "actually more predictive than an average for a Senate race," except that "at this point" has nothing to do with that, either. It may reflect a concern about the disproportion in how many people a Senate seat represents (say, between Montana and California), but individual statewide polls would be very good representations of Senate races -- to the extent that hypothetical questions are predictive. Why an "average" is relevant here ... SQUIRREL!

... The exact margin though is likely to be smaller than we thought it would be at the beginning of the cycle. It may be closer to 6 points than 8 points. And while 6 is still significantly greater than 0, it's much more doable than 8.

OK, I guess we're going to give up on the copyediting altogether.  But to the broader concern, CNN is more or less just making it up at this point. It's fun to make fun of Fox News, because Fox as a whole is so desperately a tool of the Trump wing of the Republican Party, but when it comes to straight-up reporting of survey results, Fox continues to err on the side of accurate and dispassionate. Why CNN thinks it needs to analyze public opinion by producing random, poorly punctuated babbling about innocent numbers is beyond me, but it does suggest that you should use CNN's polling analyses for entertainment value only.


* Yes, the noun modifier should be "Democrat-controlled," but the burnt hand dreads cold water.
** The phrasing is Fox's.
*** The timing could certainly affect the prospective "If the election were held today" question if it was after a primary, rather than before.
**** For convenience, it's common to use .25 as the numerator, because that produces the largest possible margin of error for any proportion from the sample. A 70-30 split on a question would produce a margin of error (95% confidence) of 3 points, rather than 3.3, with the same sample. And always report those confidence levels, kids!

Oh, stop it

Is it time to complain about the misinterpretation of public opinion data again?  Yes, and Nate Silver is the one overplaying things here.

The point of the complaint seems to be that @MeetThePress is overemphasizing the stability of Trump's approval based on a single poll finding. To be fair, yeah: Most journalists writing about survey data would do well to dial back the loud adverbs, just on principle. But on the whole, there aren't a lot of reasons to reject the null here, and when in doubt, it's always good to err on the side of not jumping up and down for the sake of jumping up and down. Let's have a look at the data and see why.

The NBC/Journal poll (n = 600 registered voters, 44% approve, 52% disapprove) is the only one I know that was in the field entirely after the (ahem) events of Tuesday. The "margin of error" at 95% confidence is +/- 4 points. As Silver notes, the approval finding is down 2 points on an NBC poll that was in the field Aug. 18-22, and with 2 points being almost exactly the standard error,* he suggests that that's a bigger finding than "stable."

I think not, for a couple of reasons. NBC/WSJ surveys from June and July showed 44% and 45% approval, respectively, so one perfectly good interpretation of the approval result is "same as in June." More importantly, though, all samples are estimates. We can't prove that the population value (the actual figure that the samples are trying to estimate) has remained stable at 45% since June, but findings of 44-45-46-44 would be a really good representation of such a reality.

It's also entirely possible (when in doubt, bet the mean) that Trump's approval has inched up and then down since June at exactly the rate the samples indicate, or even a bit more or less. The findings could reflect a change in population value from 48% to 42%. They could also all reflect a steady population value of 48% or 42%; that would be a much less likely but still nonchance result (assuming identical sample sizes, which isn't spelled out in the pdf). And any of those results could be an outlier; a 95% confidence interval tells you that one of every 20 will be, but it doesn't tell you which one (or which 20). The safest conclusion about Trump's approval among registered voters since June is: Yep, pretty stable.

True, a decline of 2 points would be "reasonably big"** if "it held across other polls," and it'd be bigger if the SE was 1.5 instead of 2, but "other polls" haven't happened yet, and I'd rather that poll stories talk about what has been found rather than what might be. And "pretty stable" is what this poll has found -- like it or not.

That, I think, is the real issue: the urge to bash the methodology when you dislike the results of a survey. That's normal journalism, in that journalism finds it hard to resist pounding the data into the shape of a story, however hard the data resist. But business as usual -- specifically, the urge to proclaim that Trump's support was "tanking" when the data said no such thing -- runs the risk of making us look both biased and stupid. I'd prefer we aim for "neither."

* For approval, the square root of ((.44)(.56))/600, if you're scoring along at home; multiply by 1.96 to get the "margin of error" at 95% confidence. For simplicity, you can use .25 for the product, which would give the maximum margin for any finding based on the whole sample.
** See what I mean about the damn adverbs?

The case of the missing rally

It wouldn't be a "fake news" story if Fox wasn't somehow involved, would it?

President Trump is creeping up in the polls now that foreign policy has seized center stage.

...  It’s hardly surprising that Trump’s approval rating is getting a little better—up six points, to 40 percent, in a new Quinnipiac poll. He launched widely praised airstrikes against Syria over a chemical weapons attack, and used the Mother of All Bombs against ISIS in Afghanistan.

Fox, of course, is hardly the only news outlet to salivate over a phrase like "mother of all bombs." Nor is it the only one to link Trump's recent bellicosity to an improvement in public assessment of his performance; that would be a "rally 'round the flag" effect, of the sort that political scientists have been tracking for decades. What's interesting here is that Fox was only just catching up with the grownup media (at top is the Washington Post, claiming a "small boost in Trump's approval rating" in an April 14 article). And what's really interesting is that the evidence shows no rally effect. This isn't just fake news; it's fake news where there should be real news, and that's news.

Read more »

Well, not really. Not at all, actually

Let's skip all the stuff about context, framing and the social construction of reality and look at a chunk of nice, old-fashioned, entirely fabricated fake news. Given those conditions, we're less interested in the creation of meaning -- after all, there is no unframed condition of news -- than in how far toward the center an out-and-out lie can spread.

The Drudge Report, as usual, isn't the originator, but it's an important indicator, because outfits (say, the Times) that don't spend much time with the drooler media do look at Drudge. Here, the vaguely policy-oriented sites (Heat Street, Campus Reform, the Education Action Group), which often wipe most of the foam off before they go out in public, are on an equal footing with sites that dispense entirely with the bib, like InfoWars and -- have you guys met ZeroHedge yet?

In the month leading up to the election on November 8th, we repeatedly demonstrated how the mainstream media polls from the likes of ABC/Washington Post, CNN and Reuters repeatedly manipulated their poll samples to engineer their desired results, namely a large Hillary Clinton lead (see "New Podesta Email Exposes Playbook For Rigging Polls Through 'Oversamples'" and "ABC/Wapo Effectively Admit To Poll Tampering As Hillary's "Lead" Shrinks To 2-Points").

Sadly, no. As in no, you didn't "demonstrate" anything, and no, that's not how sampling works, unless you have access to cheap and reliable time travel, in which case -- for an operation touting "the news that moves the markets" -- you seem to be missing a major career opportunity. As even the morons of the "unskewed polls" movement acknowledged four years ago, a difference between party identification and voter intent is not evidence of playing around with the sample. If you know the former, you can go back in time and skew the latter. Do send a postcard.

The point, again, isn't whether another huckster is playing Duke-and-Dauphin with the rubes. It's how far up the beach any specific fiction is washed before someone picks it up. Not every made-up story leads the Drudge page; the one about the pizza parlor didn't make it, but the one about Podesta drinking the chicken blood did:

And, to reiterate, the intent doesn't have to be for every story to produce an armed attack on an innocent restaurant. All that has to happen is for the story to get close enough that someone has to spend time denying it.

This is going to be a tough habit to break for news organizations that still fear they might miss the next Hoovergate, but really -- if you can't ignore fake news, at least you can run a laugh track behind it. The polls weren't rigged, the employment data weren't cooked, and you should make sport of anyone who says otherwise.

Including the president-in waiting? Well, that would be an encouraging sign.

'Well within the margin of error'

In a routine-ish campaign tale, this line stood out:

The latest polls suggest a tight race in North Carolina. Real Clear Politics polling average gives Clinton a 1.2-point lead, well within the margin of error. 

Apparently it made quite an impression elsewhere, appearing in another story and a blog post (with a touch of editing) in the same day:

The latest polls suggest a tight race in North Carolina. Real Clear Politics polling average gives Clinton a 1.2-point lead, well within the margin of error.
 
The latest polls suggest a tight race in North Carolina. The Real Clear Politics polling average gives Clinton a 1.2-point lead, well within the margin of error.

Come to that, it's had a pretty good run since convention season:

Aug. 4: Polls show a tight race in North Carolina, a key battleground. An average of recent polls by Real Clear Politics shows Clinton with a half-point lead in the state, well within the margin of error. 

Aug. 3:  Polls indicate a tight race in the state. An average of recent polls by Real Clear Politics shows Clinton with a 2-point lead in the state, well within the margin of error.

July 25: The candidates are virtually tied in North Carolina, with Clinton enjoys a 2-point advantage in the state – well within the margin of error – according to Real Clear Politics’ polling average.

July 23: Despite the spending disparity, polls show the candidates virtually tied in North Carolina. Clinton enjoys a 2-point advantage in the state, well within the margin of error, according to Real Clear Politics’ polling average.

OK, spoiler alert. Whatever you think about the phrase "within the margin of error" (which is basically journalistic gear-grinding), it doesn't apply here, because the RCP "polling average" doesn't have one. No, really: Look at the top of the chart and you'll see that RCP doesn't even pretend that the average has a "margin of error." That doesn't cause the "average" to be a meaningless number, but it is a result of the "average" being a meaningless number. Hold those thoughts for a moment while we talk about the circumstances under which journalists use adverbs and other modifiers to indicate their evaluation of the events they present to the public.

Read more »

Where the stupid starts

Wondering why The Stupid seems to be succeeding so well on the national stage? Let's ask the Wall Street Journal:

The Wall Street Journal/NBC News poll reports that 73% of the electorate believes the United States is on the wrong track, a level normally associated with national crises, such as the 2008 financial meltdown.

Reminding us, should we need reminding, that you don't have to lie about the numbers themselves to lie about public opinion. A 73% score on the bad side of the "right direction/wrong track" question is high but hardly unusual. It's not as extreme as the 79% that AP recorded at about the same time, nor is it as sunny as the 67% that Reuters found in a survey in the field July 16-20. If 73% represents the population value and neither of the other results is an outlier,* one thing that could tell you is that public opinion bounces around a lot on this question.

It can also bounce around pretty quickly, with a fairly rosy 58% saying "right direction" at the end of February 1991 (late stages of the first US-Iraq war) but "wrong track" reaching 72% in November, 78% in January 1992 and 83% in June 1992. "Wrong track" also reached 73% in June 2007 (per the Post, again) 76% in a CNN poll in August 2007 -- some months before the 2007-09 recession began. Being a grumpy lot, we were also at 79% "wrong track" in October 1990,** 77% in January 1996, 71% in June 1993 and just about equal in October 1974 (75%) and October 1973 (74%).

Well, surprise. People who are interested in public opinion try to find the stories that explain the numbers; WSJ editorial writers look for numbers to go with the stories they want to tell. So what story is being told here?

Amid the GOP convention in Cleveland and with the Democratic mother ship landing next week in Philadelphia, one has to wonder: Are the campaigns of  Donald Trump and  Hillary Clinton also on the wrong track?

As recently as a week ago, the election’s narrative was set: ... Then, just days before the GOP convention, the world snapped.

Got an idea of what comes next?
Read more »

Or not (one born every minute)

What's the latest in statistical inference there, Washington Post?

The Republican National Convention begins in Cleveland in 31 days. That means that one month from today, Republicans will (almost certainly) start the process of nominating Donald Trump as their presidential nominee to take on Hillary Clinton in the fall campaign.

That prospect looks increasingly problematic — somewhere between a stone-cold loser and a long-shot gamble. With not only the White House at stake but also Republicans' Senate majority and maybe even their House majority in real peril, the idea of nominating Trump should be cause for a growing sense of panic within GOP ranks.
No doubt they appreciate the advice. But surely there's some evidence, right?

Here's why, courtesy of a chart from RealClearPolitics detailing the polling averages for Clinton and Trump over the past three months.

Oomph.

Trump's numbers shot up in the wake of his victory in the May 3 Indiana primary, a win that effectively sealed his nomination. But as May wore on, Trump's poll numbers not only hit a wall, they began to collapse.
 
OK. Here are some suggestions if you want your analysis to be taken seriously:

1) Ignore the Real Clear Politics "polling average." It is not a meaningful number.
2) Never write paragraphs like "oomph," whether you understand the results you are describing or not.
3) If you insist on ignoring (1) and (2), at least look at the damn Y-axis.

Read more »

Please don't feed the weasels

OK, assuming that you've been keeping up with the general tone of political communication this year, you know that only one thing could make the Drudge Report happier than a picture of Hillary Clinton looking desperate (with some inept and nearly incomprehensible typography). That'd be a picture of Hillary Clinton coughing (with some inept and nearly incomprehensible typography). With that as a given, let's approach the problem from a different angle: Would the allegedly professional media please stop feeding the bottom-feeders?

Specifically, that'd be the Los Angeles Times piece linked to at Tuesday morning's Drudge homepage:

Hillary Clinton’s campaign had once hoped the California primary could be a coronation for the former secretary of State, the last major stop en route to claiming the Democratic presidential nomination.

Instead, it has turned into a dogfight with Bernie Sanders, who has been campaigning nonstop through the state. With at least one public poll showing the race now a tie, the Clinton campaign has decided to step up her appearances in the state.

Unless "public poll" means something other than "public opinion poll," or "publicly available poll" -- perhaps the Times could provide some data about the alleged poll that allegedly shows the alleged tie? Otherwise, we could fall back on the polls aggregated at Real Clear Politics, captured Tuesday morning around the same time:

Short answer, none of these five allegedly most recent polls* of the California Democratic primary show a tie. Two of them show Clinton leading by 2 points, which is not much of a lead -- it's fractionally better than even money that those differences match a real difference in the population, but you shouldn't bet on it unless randomly giving away money is your idea of a good time.**

Most of the other polls, including the one that apparently sat there for two weeks before Real Clear Politics knew it, show Clinton with a lead that is statistically significant at 95% confidence, meaning one chance in 20 that the difference in sample values doesn't reflect a similar difference in population values. The poll that doesn't is "within the margin of error," given that the "margin of error" applies to both points in the distribution," but is still almost certain to reflect a Clinton lead. Whatever "Cali" is, it is not "thisclose."

Does that mean we know who's going to win the California Democratic primary? No. Neither do you, and neither does the Los Angeles Times. It means we should be careful about how we interpret survey data.  Some things you should remember::

• Any poll result might be an outlier
• Then again, it might not

If I had to guess, meaning if I spread the results out and tried to estimate the range and midpoint of the results for Clinton and Sanders, I'd guess that the highest Clinton result might be an outlier -- meaning that of the 10 results shown in the crosstab, it seems the most likely to fall outside the normal distribution you'd expect if you drew several dozen random samples of likely California voters. But I don't know.

A corollary of that, of course, is that neither does the Los Angeles Times. What we know is that among the five results shown above, there are no ties. Of the two most recent results, one shows Clinton with a huge lead (which might or might not represent an outlier); the other shows a negligible lead (which also might or might not be an outlier).

You should at this point yawn and indicate that you're tired of polls. Fine. I'm tired of polls too, though I'm endlessly fascinated in how people make meaning out of them. My plea to the institutions that ought to know better -- the NYTs, the WashPosts, the LATimeses -- is to stop making pop fiction of them. Every time you guys turn out a fabricated drama from data that a well-trained undergraduate would question, you enable the genuine sleazebags like Drudge to do their thing.

Please stop. Please. Democracy is actually at stake.

* As of Tuesday evening, RCP was showing four polls: the top three above and one that was in the field May 4-16, of 694 likely voters, showing Clinton with 51% to Sanders' 38%. RCP does not provide a margin of sampling error for that poll, which should raise some doubt in your mind about RCP's general competence at reporting quantitative data. (The maximum margin of sampling error at 95% confidence for a random sample of n = 694 is -- who wants to be first in the comments?)
** Everybody has a hobby. 
*** Pay attention, FiveThirtyEight 

Making stuff up with numbers

Nothing wrong with the headline, as long as you don't expect headlines to be true or anything. The race might indeed be "tightening," but none of the evidence here supports such a conclusion -- mostly because the "polls" don't show anyone "tied" with anyone else. Fox isn't the only cable network to get this point wrong, but it appears to be the most consistent in lying about numbers for partisan gain, so let's have a look:

Two polls released Sunday show Democratic presidential front-runner Hillary Clinton tied with presumptive Republican nominee Donald Trump in a likely general election race, after having a double-digit lead just months ago.
  
Clinton leads Trump 46-to-43 percent in a NBC News/Wall Street Journal poll, compared to a similar one in April in which Clinton had an 11-point lead.

Odd. A 3-point lead wasn't a "tie" last week, when Fox reported on its own poll:

Donald Trump tops Hillary Clinton in a hypothetical head-to-head matchup, according to a new Fox News Poll that also finds majorities of voters feel both frontrunners lack strong moral values and will say anything to get elected.

Trump has a 45-42 percent edge over Clinton, if the presidential election were held today.  That’s within the poll’s margin of sampling error.  Last month, Clinton was up by 48-41 percent (April 2016).

Hmm. Let's return to Sunday's story for a moment:

Earlier Sunday morning, a Washington Post/ABC News poll showed voters favored Trump over Clinton 46-to-44 percent. The numbers also show Clinton losing an identical 11-point lead since earlier this spring.

Both polls were within the statistical margin of error, which means Clinton and Trump are essentially tied.

Well, no. It doesn't mean that. Not that the misperception isn't widely shared -- for example, by NBC itself (here, in its MSNBC guise):

Hillary Clinton’s advantage over Donald Trump has narrowed to just three points — resulting in a dead-heat general-election contest with more than five months to go until November, according to a new national NBC News/Wall Street Journal poll.

Clinton, who remains a heavy favorite to win the Democrat nomination, leads the presumptive GOP nominee 46 percent to 43 percent among registered voters, a difference that is within the poll’s margin of error of plus-or-minus 3.1 percentage points.

NBC isn't done with the stupidz:

Read more »