Unraveling the 2014 local election changes in the share of the vote: who suffered most from UKIP?

A naïve glance at the overall changes in the share of the vote since 2010 in the table below suggests that UKIP’s big gains came as the expense of the biggest losers, the Liberal Democrats.

	Change % vote since 2010	Change % vote since 2012
Con	-6	-0.2
Lab	+5	-8
LD	-14	-3
UKIP	+13	+10
Grn	+4	+2
BNP	-2	-0.2
Oth	+0	-0.6

(Overall changes in the share of the vote across the BBC’s 1008 Keywards.)

This is clearly one of those cases where the net changes in the share of the vote can be misleading.  The opinion polls have commonly told us that Labour have benefitted primarily from the collapse of the Liberal Democrats, whereas the UKIP surge has come mostly from the Tories. 

If all the Lib Dem losses since 2010 were to Labour then they should be up 14 not just 5. The Green 4 point gain is likely to have largely come from the LDs, but that would still leave Labour 5 points short of where they would have been if they were simply benefitting from the Lib Dem losses.

Conversely, on the right of the political spectrum, if, apart from 2 points from the BNP, all of UKIP’s 13 point rise since 2010 were from the Tories, then Conservative would have been down by 11 points, not 6 they were actually down by. This discrepancy of 5 points corresponds to the 5 points Labour seem to have been short by in the above paragraph.

On this rough reckoning then, UKIP took 5 points from Labour, and 6 points from the Conservatives.  So, by comparison with 2010, it looks like Farage hurt both major parties roughly equally, not disproportionately from the Tories.

The story about change since 2012 also in the table is also revealing here.  That shows much more clearly how Labour have fallen and UKIP risen in local election vote shares over the last two years.  What seems to have happened is that between 2010 and 2012 UKIP took votes mainly from the Conservatives, but between 2012 and 2014 they have had more success in attracting Labour voters. The net effect is that the UKIP rise from 2010 to 2014 has been at similar expense to Labour and the Conservatives.

This is certainly a story that fits the main pattern of change in the general election vote intention opinion polls. The narrowing of the Labour lead over the past two years has been mainly due to a Labour fall and UKIP rise.

Another way of looking at this issue is to consider who suffered most in the wards where the UKIP vote share went up most.  In BBC Keywards fought by all four parties, the correlation between the change in the UKIP share of the vote since 2010 and that for the Conservatives was very similar to that between UKIP and Labour: both close to -0.3.  This provides further evidence that UKIP hurt Labour by about the same amount as they hurt the Conservatives.

This pattern is also visible in the main source of regional variation.  Across wards fought by all three main Westminster parties, UKIP were up by just 6 points in London, but by 14 points elsewhere in England: an 8 point difference.  The corresponding (partially) compensating London/elsewhere differences in the performances of the Conservatives, Labour and Liberal Democrats were 2.0, 2.6 and 1.4 points respectively.  These figures not only reinforce the picture of Labour suffering similarly if not more than the Tories from UKIP, but they also show an (albeit smaller) impact on the Liberal Democrats.

So much for votes. What about seats? Among the 53 UKIP gains in the BBC Keywards, 22 came from the Conservatives, 19 from Labour and 12 from the Lib Dems.  So a similar pattern again. 

Acknowledgements: Thanks to the BBC, John Curtice, Rob Ford, Jon Mellon and Rosie Shorrocks for their help with both data and analysis.

What do the 2014 local election results mean for next year’s general election?

The Labour lead of 2 percentage points in the BBC’s projected national share of the local election vote (PNS) is too narrow a lead for Labour to suggest they will be ahead in a general election next year. 

Since it was first calculated (in 1983) oppositions have usually had bigger leads in the PNS in the year before a general election, but they still went on to lose the next election.  The changes of government (in 1997 and 2010) were preceded by very big leads for the opposition in the PNS (16 points for Labour in 1996 and a 15 point lead for the Tories in 2009).

The message from the PNS is in line with general election vote intention in the opinion polls, which have on average also shown a narrow lead for Labour recently.

Even though their PNS is lower than the 23% they scored last year, the UKIP challenge remains significant. But since they seem to be hitting both main parties roughly equally, it is perhaps more important to think about both the impact on the major parties and it’s distribution across the country.

A uniform swing projection of the House of Commons from the PNS suggests that UKIP would not win any seats. This is partly because in 2010 their share was very evenly distributed.

There were some signs last year of UKIP’s vote becoming more concentrated in areas with more older people, fewer graduates and fewer ethnic minorities. 

UKIP share became still more variable this year. They went up more where they started strongest. So they are building up strong support in particular places, which is what they need to do to win seats in Westminster.  Indeed UKIP won the share of the vote across all the wards in the Great Grimsby constituency. This was the seat that Robert Ford and Matthew Goodwin predicted would be the most fertile territory for a UKIP general election victory.

(Rotherham has also been mentioned as an effective UKIP constituency win, but it was not part of the BBC keywards data collection so I’m not in a position to be able to confirm.)

Moreover, there are several seats where, based on the results of this week’s local elections UKIP would not be far behind the winning party. These include Portsmouth North (5.9% behind Con), Portsmouth South (6.1% behind LD), Southport (9.3% behind LD), Bradford South (9.4% behind Lab). 

Just as in local elections, UKIP are more likely to win seats in the Commons where the share of the vote between the three main parties is more evenly divided. In a general election if it becomes apparent that UKIP are a serious challengers this will lead to the kind of attention that might generate strong efforts to encourage tactical voting both for and against UKIP. So it may be that UKIP will find it difficult to divide and conquer.

Acknowledgements: Thanks to the BBC, John Curtice, Rob Ford, Jon Mellon and Rosie Shorrocks for their help with both data and analysis.

BBC Projected National Share of Vote 2014

The BBC Projected National Share of the vote just announced is Con 29, Lab 31, LD 13, UKIP 17, Others 10.

The PNS is an attempt to estimate what the share of the vote would be if the whole of GB had local elections and if the three main Westminster parties had fielded candidates in all wards, as they do in general elections. For more details see here and here.

Graphs below show how it compares with previous figures and polls historically.

Locals seats forecast from national polls 2014

Last year I published details of a basic model for forecasting the number of local council seats won from national opinion polls here. The basic idea is to estimate change in seats since the last time they were fought (four years ago) as a function of change in the polls over that time.  Despite the massive increase in UKIP candidature last year and their remarkable achievement of a Projected National Share (PNS) of the vote of 23%, the model still performed relatively well. In part because the even distribution of their vote meant UKIP only won 147 seats. 

The following table shows the actual number of gains and losses in 2013, and both my model estimates and the Rallings and Thrasher forecast.

	Actual	Estimate	Range	Rallings & Thrasher
Con	-355	-205	-295 to -115	-310
Lab	+291	+260	+160 to +355	+350
LD	-124	-190	-85 to -300	-130

Having updated the model, the estimated gains and losses of seats in tomorrow's local elections are as shown in the table below.

	Estimate	Range	Rallings & Thrasher
Con	-130	-235 to -30	-220
Lab	+130	+30 to +220	+490
LD	-400	-570 to -230	-350

The Rallings and Thrasher forecasts come from their PSA briefing.  The pattern of who's expected to be up and who down is straightforward. The seats up this year were last fought on the same day as the 2010 general election; Labour have recovered since then while the governing parties have gone down in the polls. The Liberal Democrat drop is particularly dramatic.

The difference between my model and the Rallings and Thrasher forecast is that for the two main parties the changes are much more muted in my model, especially for Labour.  This maybe partly because they are also predicting UKIP +80, but that still leaves over 300 seat discrepancy to explain. (My model won't be capable of sensibly estimating UKIP seats till 2016: four years after they first fought a large majority of wards in local elections.)

It also doesn't look like the discrepancy is due to differences in estimates for the overall share of the vote. The Rallings and Thrasher forecast 2014 National Equivalent Vote (NEV, similar to PNS) is Con 30, Lab 33, LD 14, UKIP 16, Other 7.  If anything this shows smaller losses and gains since 2010 than do the opinion polls I'm using.  

The graph below shows both the polls and the PNS over time. The rise of UKIP last year led to bigger drops in the PNS for Labour and the Conservatives than might otherwise be expected.  Given the changes in the polls since 2013 and the concurrent Euro elections, the PNS for Con, Lab and UKIP could be close, with UKIP likely to do better than the 16% Rallings and Thrasher have forecast for them.

Returning to the seats forecasts, the differences between the two are most likely to be due to Rallings & Thrasher more careful ward level estimates. My model is a national level only one, estimating the overall share of seats won by each party. Their approach is more detailed and should work better. 

Acknowledgements: Thanks to David Cowling, John Curtice, and Michael Thrasher for help with the data.

Reasons for changing the forecasting model: a response to The Economist

The Economist magazine have published a great article on the difficulties of predicting the next election. It makes lots of good points well and kindly covers my 2015 general election forecasting model. But I was bemused by the line that reads, "Within months he had published a revised model: polls had not adhered to the original one."

I did revise my October 2013 model in February 2014, but not because the polls had not adhered to the original one. They have kindly published a correction. The rest of this post sets out why adherence of polls to the model has be adequate and explains the actual basis for the change in the model, before finally noting that adherence to the polls might weaken close to the European Parliament and local council elections.

As I wrote last week here, there has been very little change in the polls or the forecasts since October, especially relative the to prediction interval for the forecast. Given the polls, this would be true for any model of this kind at this stage in the election cycle. Without dramatic swings in the polls, we are too far from the election and there is too much variation in the historical record to be able to say that deviations from the central estimates of any such forecasting model are so big as to cast doubt on any particular model.

Only today, have I looked at how both models changed between the first published forecast on the website (25th Oct 2013) and last published forecast for the old model before switching to the new one (31st Jan 2014).  While the two models have substantially different predictions the change in the forecast share of the vote for both models over this period was the same: Con +1, Lab -1 and less than a percentage point change for LD. 

So not only was there was no reason to switch models on the basis of adherence to the polls, but the trajectory of the polls has been as close to that predicted by both models as could reasonably be expected.

What prompted the change of the model was explained at the time here, “The change in the methodology is due to a couple of the comments which prompted me to reconsider modelling reversion to the last election, which was an approach I had neglected too quickly.” The justification for the switch of model in the February working paper is better performance at predicting past election results: nothing to do with polls in this cycle. 

If and when I do change the model again (I have said I probably would) it will be an attempt to better reflect the most important aspects of the historical record. Forecasting the current election cycle does require some judgement calls which should depend on current circumstances. But the changes in the polls over the last year do not tell us much as to what kind of model to prefer.

All that said, as we approach the European Parliament and local council elections we are likely to see general election vote intention in the opinion polls change. The Conservatives dropped and UKIP rose in the polls in run up to last year’s local elections. Similarly in the month before the 2009 European and local elections, support for the government dropped while UKIP and other minor party support rose dramatically, not just in European Parliament vote intention but in general election polls too. The MPs’ expenses crisis doubtless contributed to the 2009 changes and this pattern was much more muted in 2004. In all of these cases there was some reversion back to pre-campaign levels afterwards.

This pattern does not always hold, certainly not for all local elections. But if this kind of phenomenon happens again this year then my forecasting model will behave strangely but revert back to normal in a couple of months time.

How the 2015 general election forecast probabilities have changed thus far and why

The graph below shows how my 2015 general election forecast probabilities have changed since October last year.*  The blue line shows the probability that the Conservatives will have the largest number of seats. The corresponding red line for Labour is just the mirror image. Also included in the graph is the probability of a hung parliament.

There hasn't been much change, and the most important thing to note is that there is a lot of uncertainty. The Conservatives have the edge but 59:41 is not a whole lot better than 50:50.

What little change there has been since October has to Labour's advantage. The estimated probability of Labour being the largest party has risen from 36% from 41%. There has also been an increase in the chances of a hung parliament from 40% to 48%.

These changes in forecast probabilities are a function of both forecast shares of the vote and the estimated uncertainty in those shares.  The uncertainty has declined a little since October, but is still huge. The changes in the probabilities are mostly driven by changes in the forecast shares which are shown in the graph below.

Again the main impression is of little change, both in the polls (pale lines) and the forecasts (dark lines). It is still the case that the model predicts a reversal of positions for the Conservatives and Labour and a recovery for the Liberal Democrats. 

While Labour has lost about three points in the polls since October, the two governing parties haven't made any progress at all. Labour losses have largely been matched by a rise in the UKIP share.

The Conservative's failure to improve their standing in the polls since October means their forecast general election share has dropped from 38% to 36% and that is the main reason why the forecast probability that they will win has dropped. 

Since the forecast Conservative lead over Labour in 2015 has also narrowed (to 5 percentage points), the probability of a hung parliament has increased.

Change in the polls relative to that predicted by the forecasting model can be thought of as a measure of party performance. It tells us whether the government is making the kind of recovery previous governments did at this stage and whether the opposition is managing to hold on to mid-term gains better or worse than previous oppositions.  While Labour are now in the polls exactly where the model and the October polls suggested they should be, the governing parties are each a couple of points behind. So the government has not since October achieved the recovery from mid-term blues that history suggests they ought to have done.  

But the difference between observed and predicted change over the last seven months has been small and dwarfed by the forecast uncertainty for both that period and for the year to come. So we are not much clearer about the likely outcome of the general election now than we were in October. The Conservatives still have the edge but there is still everything to play for.

*Methodological note: The graphs here use the revised method from the February 2014 working paper. I have gone back and reconstructed the series of forecasts from October 2013 to February 2014 with the February method rather than use the published forecasts. This is not to try to change history but just to compare like with like. The original October 2013 working paper is here. 

A long range forecast for the UKIP share of the vote at the 2015 general election

Previous posts at this blog introduce my long-range general election forecasting model, which is updated on Fridays at http://users.ox.ac.uk/~nuff0084/ge15forecast/.

Thus far I haven't published a forecast of the UKIP share of vote because the party hasn't been competing in general elections for long enough to build a proper statistical regression model of the relationship between their support in the polls and their votes at elections, as for the three main parties. But the forecast has nonetheless been suggesting a combined 'Others' share of the vote around 17% at the next election.

This estimate is naturally heavily dependent of the current fortunes of UKIP in the opinion polls.  So there is a case for estimating the UKIP share of the vote in 2015 using their current share of the combined Others share in the polls.  This will now be a regular feature of the published forecast.

As you can see from the current forecast below, UKIP are at 12% in the current UKpollingreport.co.uk polling average. This is 60% of the combined Others total of 20 percentage points.  Since the current forecast for all Others combined is 17.0%, the UKIP forecast share is 0.6*17 = 10.2%.

I still haven't decided how best to estimate prediction intervals for the Others and UKIP, but they are likely to be of a similar size to those for the main parties, i.e. plus or minus 8 percentage points.

Despite forecasting a substantial increase in the share of the vote for UKIP, the model is not forecasting any seats for UKIP in 2015. This is despite attributing all of the increase in the Others share to UKIP wherever it was the largest Other party in 2010.  This reflects the very flat distribution of the UKIP vote across constituencies in 2010.

Whether the UKIP vote will be similarly evenly spread in 2015, or whether they will be able to mount successful campaigns in selected seats remains to be seen.  It is certainly something that can't be predicted this far from the election with the statistical technology this forecasting model is based on.

Date of forecast: 04.04.2014
Days till the election: 398

Inputted current average poll shares 
Con : 33
Lab : 37
LD  : 10
Oth : 20
UKIP: 12 so UKIP share of combined Others vote is .6

Forecast Election Day Shares with 95% Prediction Intervals
Con : 36.9 plus or minus 8.4 i.e. between 29 and 45
Lab : 31.8 plus or minus 6.3 i.e. between 26 and 38
LD  : 14.3 plus or minus 9.9 i.e. between 4 and 24
Implied point estimate shares for:
   Others combined: 17.0
   UKIP           : 10.2 (based on share of combined Others vote.)

Forecast Election Day Seats
Con : 308
Lab : 283
LD  : 31
Con largest party, but short of a majority by 18

Forecast Election Day Seats with approximate 95% Prediction Intervals
Assuming LD share at 14.3 and Other share at 17 and allowing Con and Lab to vary as per intervals above.
Con between 218 and 413
Lab between 185 and 368
LD between 25 and 39

Approximate probabilities of key outcomes
Pr(Con largest party) = 60%
Pr(Lab largest party) = 40%
Pr(Con majority) = 35%
Pr(Lab majority) = 19%
Pr(Hung parliament) = 46%
Pr(Hung parliament with Con largest party) = 25%
Pr(Hung parliament with Lab largest party) = 21%