Revised long-range forecasting method for a 2015 British General Election

I have revised my 2015 forecasting methodology. There is a new working paper here. The old one from October 2013 is still here, and there are past blog posts here and here that provide more background. The current forecast is below and will be updated weekly here. 

The new method averages forecasts from two different forecasting models: the old model in which parties have a tendency to move towards their historical average level of support, and a new model in which parties tend to move towards the last election result. (Technically, these are two different regression-to-the-mean effects but with different means.) Both models perform equally well for previous elections and are relatively similar for the Conservatives and Labour, but they have dramatically different predictions for the Liberal Democrats. Allowing for reversion to a long-run level of support produces a low Lib Dem forecast, while a model based on reversion to the previous election result generates a high forecast for them. Without any strong methodological reason to prefer one over the other it is arguably best to average over them.

Compared with using the old model alone, the new combined method raises the predicted election-day share by 4 percentage points for the Liberal Democrats, lowers the Conservative forecast by 2 points, with little change for Labour.  The net effect on the probabilities is to reduce the probability of the Conservatives being the largest party in parliament from 60% to 56% and increase the probability of a hung parliament from 32% to 44% (because of the higher Lib Dem share and lower Tory lead). 

Compared with October 2013, when I first started publishing a forecast, using the new method the chances of a Tory lead in seats has dropped by 8 percentage points and chances of a hung parliament have increased by the same amount (full details in the paper). The positions of the parties in the polls have barely changed over this period. The change in the forecast reflects the greater proximity to the election. Since there has been no Conservative recovery over the last four months, the expected eventual recovery is now smaller than previously forecast. By this measure, Labour have done well to maintain their lead since the middle of last year. 

The change in the forecast does not mean that the old forecast was wrong or invalid. The new forecast is based in part on the old model and the bounds of the new prediction mostly overlap with the bounds from the October forecast. It is more appropriate to think of the changes as updates in light of more information. The historical relationship between polls and votes still suggests the Conservatives are more likely than not to emerge ahead of Labour in votes at the election. Although because of the bias in the electoral system, the outcome in seats is too close to call.

Thanks to all those who (politely) commented on the previous paper or blog post. The change in the methodology is due to a couple of the comments which prompted me to reconsider modelling reversion to the last election, which was an approach I had neglected too quickly. I think the method is better for it so I’m very grateful. 

Most comments on the old model focused either on UKIP or the effect of coalition government on the chances of a Liberal Democrat recovery. The latter is something I agree is practically impossible to model properly in this framework because this is the first time since 1945 that the Liberals have been in government. The rise of UKIP is a major development and although it is not formally modelled the forecast of the combined Other share of the vote mostly reflects UKIP’s high standing in the polls. So UKIP have not been ignored and they are expected to do unprecedentedly well in 2015. 

I will continue to consider modifications to the methodology, so comments are still very welcome. I think the main thing that is missing from the model at the moment is an adjustment for first-term incumbency, which should help the Conservatives, but that will take some time to estimate. Much as I’m sympathetic to some of the comments about things that might mean this time is different, there is not much I can do about them in this framework which relies on understanding how polls translated into votes are previous elections. However, the prediction intervals on the forecast are wide because of all of the idiosyncrasies of previous elections. Whether those for this election are really much different will not be known until after the election. 

Current forecast from the new methodology:

Date of forecast: 10.2.2014
Days till the election: 451

Inputted current average poll shares 
Con : 32
Lab : 39
LD  : 9

Forecast Election Day Shares with 95% Prediction Intervals
Con : 36.8 plus or minus 8.8 i.e. between 28 and 46
Lab : 32.5 plus or minus 6.6 i.e. between 26 and 39
LD  : 14 plus or minus 10.8 i.e. between 3 and 25

Implied point estimate for Others combined: 16.7

Forecast Election Day Seats
Con : 304
Lab : 289
LD  : 30
Con largest party, but short of a majority by 22

Forecast Election Day Seats with approximate 95% Prediction Intervals
Assuming LD share at 14 and Other share at 16.7 and allowing Con and Lab to vary as per intervals above.
Con between 211 and 413
Lab between 186 and 375
LD between 23 and 38

Approximate probabilities of key outcomes
Pr(Con largest party) = 56%
Pr(Lab largest party) = 44%
Pr(Con majority) = 33%
Pr(Lab majority) = 23%
Pr(Hung parliament) = 44%
Pr(Hung parliament with Con largest party) = 23%
Pr(Hung parliament with Lab largest party) = 21%