The ten most surprising Senate races since 1972


Which races would The Economist’s new model have had most trouble predicting?

United States

Politics can be unpredictable. As lucky as psephologists are to have bountiful polls and fundraising numbers to forecast election results, models can still err in predicting them for a number of reasons. Notable sources of error include systematic biases in polls or a shift in the race in the final week of the campaign, after most pollsters have finished their surveying. Both of those help to explain the infamous misfire in 2016. But there was a time, before our poll-obsessed contemporary era, when errors were mainly due to a lack of data.

The Economist’s brand-new forecasting model for the US Senate has helped to unearth historical upsets. Retrofitted it to past elections, it has successfully predicted 94% of Senate races since 2000. However, in contests without polling data or fundraising numbers, our model has little to chew on. It is much less accurate for elections before the turn of the current century than in the years since. We would have been able to predict the outcome of just 82% of contests between 1972 and 1998.

Nine of the top ten misfires in our forecast came in Senate elections held before 2000. In the most extreme cases, candidates to whom we gave only a 2-3% chance of winning ended up pulling off an upset. (Although these figures are striking, it is notable that our forecast is still well calibrated. When we say that candidates have, say, a 5% chance of winning, one in 20 of such candidates does indeed win.)

These misfires teach us several things about American politics (and a few about forecasting). First, Senate races in the 1970s were only weakly predictable by the so-called political “fundamentals” of the election—namely, how the state broke in the Senate race last time and whether an incumbent was on the ballot. Second, the availability of fundraising data in the 1980s increased our ability to gauge voters’ enthusiasm for challengers. And third, modern levels of partisanship have made it much easier to predict the outcomes of Senate races from the most recent presidential vote in a given state.

Take our model’s biggest-ever miss, in New Hampshire in 1978. Thomas McIntyre, the incumbent Democratic senator, had been re-elected twice and regularly outperformed Democratic presidential candidates. In 1972 when George McGovern, the Democratic nominee, only won 35% of granite-staters, McIntyre was re-elected with 57% of the vote. Our model would have put his chance of defeat in 1978 at just 3% (see chart). Yet the Republican, Gordon Humphrey, beat him.

Elections since Mr McIntyre’s defeat have been more predictable. On average, the margin of error on our Senate forecast’s estimate of the Democratic share of the vote in any given state has shrunk from 30 percentage points in 1972 to 18 points today. And for states that are heavily polled the uncertainty intervals can be as small as seven or eight points. For example, our model thought Ron Johnson, who was elected as a Republican senator for Wisconsin in 2010, only had a 6% chance of overcoming a seven-point deficit in a state that had voted for Democratic presidential candidates since 1988. The Economist’s new Senate model reminds us that even in our hyper-polarised era, unpredictable things can happen.

Editor’s note: This article first appeared in our weekly “Checks and Balance” newsletter on American politics. You can sign up to receive it here.

Reuse this contentThe Trust Project

View original post