Friday, 4 April 2014

A long range forecast for the UKIP share of the vote at the 2015 general election

Previous posts at this blog introduce my long-range general election forecasting model, which is updated on Fridays at http://users.ox.ac.uk/~nuff0084/ge15forecast/.

Thus far I haven't published a forecast of the UKIP share of vote because the party hasn't been competing in general elections for long enough to build a proper statistical regression model of the relationship between their support in the polls and their votes at elections, as for the three main parties. But the forecast has nonetheless been suggesting a combined 'Others' share of the vote around 17% at the next election.

This estimate is naturally heavily dependent of the current fortunes of UKIP in the opinion polls.  So there is a case for estimating the UKIP share of the vote in 2015 using their current share of the combined Others share in the polls.  This will now be a regular feature of the published forecast.

As you can see from the current forecast below, UKIP are at 12% in the current UKpollingreport.co.uk polling average. This is 60% of the combined Others total of 20 percentage points.  Since the current forecast for all Others combined is 17.0%, the UKIP forecast share is 0.6*17 = 10.2%.

I still haven't decided how best to estimate prediction intervals for the Others and UKIP, but they are likely to be of a similar size to those for the main parties, i.e. plus or minus 8 percentage points.

Despite forecasting a substantial increase in the share of the vote for UKIP, the model is not forecasting any seats for UKIP in 2015. This is despite attributing all of the increase in the Others share to UKIP wherever it was the largest Other party in 2010.  This reflects the very flat distribution of the UKIP vote across constituencies in 2010.

Whether the UKIP vote will be similarly evenly spread in 2015, or whether they will be able to mount successful campaigns in selected seats remains to be seen.  It is certainly something that can't be predicted this far from the election with the statistical technology this forecasting model is based on.

Date of forecast: 04.04.2014
Days till the election: 398

Inputted current average poll shares 
Con : 33
Lab : 37
LD  : 10
Oth : 20
UKIP: 12 so UKIP share of combined Others vote is .6

Forecast Election Day Shares with 95% Prediction Intervals
Con : 36.9 plus or minus 8.4 i.e. between 29 and 45
Lab : 31.8 plus or minus 6.3 i.e. between 26 and 38
LD  : 14.3 plus or minus 9.9 i.e. between 4 and 24
Implied point estimate shares for:
   Others combined: 17.0
   UKIP           : 10.2 (based on share of combined Others vote.)

Forecast Election Day Seats
Con : 308
Lab : 283
LD  : 31
Con largest party, but short of a majority by 18

Forecast Election Day Seats with approximate 95% Prediction Intervals
Assuming LD share at 14.3 and Other share at 17 and allowing Con and Lab to vary as per intervals above.
Con between 218 and 413
Lab between 185 and 368
LD between 25 and 39

Approximate probabilities of key outcomes
Pr(Con largest party) = 60%
Pr(Lab largest party) = 40%
Pr(Con majority) = 35%
Pr(Lab majority) = 19%
Pr(Hung parliament) = 46%
Pr(Hung parliament with Con largest party) = 25%
Pr(Hung parliament with Lab largest party) = 21%

Monday, 10 February 2014

Revised long-range forecasting method for a 2015 British General Election

I have revised my 2015 forecasting methodology. There is a new working paper here. The old one from October 2013 is still here, and there are past blog posts here and here that provide more background. The current forecast is below and will be updated weekly here

The new method averages forecasts from two different forecasting models: the old model in which parties have a tendency to move towards their historical average level of support, and a new model in which parties tend to move towards the last election result. (Technically, these are two different regression-to-the-mean effects but with different means.) Both models perform equally well for previous elections and are relatively similar for the Conservatives and Labour, but they have dramatically different predictions for the Liberal Democrats. Allowing for reversion to a long-run level of support produces a low Lib Dem forecast, while a model based on reversion to the previous election result generates a high forecast for them. Without any strong methodological reason to prefer one over the other it is arguably best to average over them.

Compared with using the old model alone, the new combined method raises the predicted election-day share by 4 percentage points for the Liberal Democrats, lowers the Conservative forecast by 2 points, with little change for Labour.  The net effect on the probabilities is to reduce the probability of the Conservatives being the largest party in parliament from 60% to 56% and increase the probability of a hung parliament from 32% to 44% (because of the higher Lib Dem share and lower Tory lead). 

Compared with October 2013, when I first started publishing a forecast, using the new method the chances of a Tory lead in seats has dropped by 8 percentage points and chances of a hung parliament have increased by the same amount (full details in the paper). The positions of the parties in the polls have barely changed over this period. The change in the forecast reflects the greater proximity to the election. Since there has been no Conservative recovery over the last four months, the expected eventual recovery is now smaller than previously forecast. By this measure, Labour have done well to maintain their lead since the middle of last year. 

The change in the forecast does not mean that the old forecast was wrong or invalid. The new forecast is based in part on the old model and the bounds of the new prediction mostly overlap with the bounds from the October forecast. It is more appropriate to think of the changes as updates in light of more information. The historical relationship between polls and votes still suggests the Conservatives are more likely than not to emerge ahead of Labour in votes at the election. Although because of the bias in the electoral system, the outcome in seats is too close to call.

Thanks to all those who (politely) commented on the previous paper or blog post. The change in the methodology is due to a couple of the comments which prompted me to reconsider modelling reversion to the last election, which was an approach I had neglected too quickly. I think the method is better for it so I’m very grateful. 

Most comments on the old model focused either on UKIP or the effect of coalition government on the chances of a Liberal Democrat recovery. The latter is something I agree is practically impossible to model properly in this framework because this is the first time since 1945 that the Liberals have been in government. The rise of UKIP is a major development and although it is not formally modelled the forecast of the combined Other share of the vote mostly reflects UKIP’s high standing in the polls. So UKIP have not been ignored and they are expected to do unprecedentedly well in 2015. 

I will continue to consider modifications to the methodology, so comments are still very welcome. I think the main thing that is missing from the model at the moment is an adjustment for first-term incumbency, which should help the Conservatives, but that will take some time to estimate. Much as I’m sympathetic to some of the comments about things that might mean this time is different, there is not much I can do about them in this framework which relies on understanding how polls translated into votes are previous elections. However, the prediction intervals on the forecast are wide because of all of the idiosyncrasies of previous elections. Whether those for this election are really much different will not be known until after the election. 

Current forecast from the new methodology:

Date of forecast: 10.2.2014
Days till the election: 451

Inputted current average poll shares 
Con : 32
Lab : 39
LD  : 9

Forecast Election Day Shares with 95% Prediction Intervals
Con : 36.8 plus or minus 8.8 i.e. between 28 and 46
Lab : 32.5 plus or minus 6.6 i.e. between 26 and 39
LD  : 14 plus or minus 10.8 i.e. between 3 and 25

Implied point estimate for Others combined: 16.7

Forecast Election Day Seats
Con : 304
Lab : 289
LD  : 30
Con largest party, but short of a majority by 22

Forecast Election Day Seats with approximate 95% Prediction Intervals
Assuming LD share at 14 and Other share at 16.7 and allowing Con and Lab to vary as per intervals above.
Con between 211 and 413
Lab between 186 and 375
LD between 23 and 38

Approximate probabilities of key outcomes
Pr(Con largest party) = 56%
Pr(Lab largest party) = 44%
Pr(Con majority) = 33%
Pr(Lab majority) = 23%
Pr(Hung parliament) = 44%
Pr(Hung parliament with Con largest party) = 23%
Pr(Hung parliament with Lab largest party) = 21%


Friday, 25 October 2013

Initial responses to comments so far on the long-range election forecasting post

Thanks for all the various comments on my initial post about long-range forecasting from historical polls and votes relationship http://electionsetc.blogspot.co.uk/2013/10/a-long-range-forecast-for-2015-british.html

Here are some initial responses to comments and questions that have been made on Twitte and on the blog. I've grouped them according to theme and pasted them unedited. I haven't copied everything but I have tried to take examples of all points/queries. Sorry if I've inadvertently left something out but I haven't deliberately done so.

The response to the first set is an apology for something I should have sorted out and will try to resolve. The rest are attempts at explanation, clarification and hopefully helpful responses. Apart from the first three the topics are in no particular order.

Incompatible probabilities:
  • Pr(Lab majority) = 15% & Pr(Lab largest party) = 12% are not compatible.
  • How can Lab have a bigger chance of a majority than being the largest party?
Fair cop. These figures are logically incompatible. Clearly I need to revise the method to ensure these are fully consistent, and I will do. For computational reasons the method does not currently estimate a full uncertainty distribution over all possible seat outcomes which is necessary for guaranteed strictly consistent estimates. If you read the paper you'll see that, at present, the probabilities for Con majority, Lab majority and difference between Con and Lab are generated from three different assumed normal distributions from the prediction intervals (for Con seats, Lab seats and the difference respectively). These are not strictly defined to be fully compatible. Much of the time this shouldn't matter but did on in this situation and I should have at least flagged this up and discussed it or solved the problem before publishing. Sorry.

It might take a bit of time to resolve the problem but unlikely to make much different to the main story - that the probabilities for Labour largest or Labour majority are small.  Most likely direction of change with a different method is that the probability of Lab largest party will be revised up, but I would be surprised if it became more than 25%. 

All the probabilities come from approximate methods so best just look at them as rough estimates. 

Conflict of interest?
  • Someone got paid from CCHQ?
No - I'm not paid by any political party and have never made a political bet. No conflict of interest issues here at all.

What about UKIP?
  • Seems odd that there's no mention of UKIP. Not convinced this is best use of regression ever.
  • Does this include #UKIP causing Conservatives in marginal seats losing to Labour?
  • No Ukip MPs?
  • No serious prediction can be made without factoring in Ukip They will take Labour as well as Conservative votes
  • This model is flawed as it takes no account of UKIP's rise in popularity (even allowing for a possible tail off at the GE).
UKIP are factored in indirectly but as a young party we can't model their historical experience of polls and election results in the same way as for the main parties. UKIP current standing is reflected in the input share of the vote, which has the Others on 18. Predicted Other share in 2015 is 16.2, which you could say is likely to be mostly UKIP. Apart from Nationalists, the seats prediction assumes that the largest other party in any constituency will get the full amount of the predicted 2010-15 change in Other share. Despite this, UKIP are still not predicted to win any seats - they are just too far behind.

The model doesn't allow for UKIP to benefit disproportionately from any one party. I can see this is a concern but note that Rob Ford, Matthew Goodwin, myself and others have been pointing out that UKIP take votes from Labour as well as the Tories, so the differential impact on Tories may be much more muted than many assume. 

Prediction is implausible?
  • Forgive my scepticism. No-one has polled over 40% in an election for 12 yrs, and the Tories haven't for 21.
  • those Con + Lab vote shares would be mirror of Blair vs Hague in 2001 
  • All 3 points seem logical and v probable.. but direction of travel seems too drastic to take seriously.
I agree that the prediction is remarkable, but think it is worth seeing what the data say. The patterns in previous cycles are surprisingly consistent. Paper discusses idea of weighting more recent elections more but that doesn't improve out-of-sample prediction substantially. I don't think it is a good idea to try to find the single previous election that seems to best characterise what you think is going on and rely on that. Better to pool information from a set of previous elections.

This time is different because of the coalition
  • History of Tory incumbency barely relevant because Coalition are incumbents not Tories.
  • Surely many historical precedents are not valid because we are in a unique situation - there will be no incumbent government defending its position. All parties will be campaigning on the basis of change.
  • Nor does it allow for the fact that a significant number of unhappy Lib-Dem voters will be registering their dis-satisfaction with the Coalition & voting Labour. As Mike Smithson says, this will be a unique election. Such generalised models based on previous voting patterns are, therefore, unlikely to be correct.
Maybe there are good reasons to believe that things are not likely to be so different. Research on coalition government electoral performance cross-nationally by myself and others suggests that the Tories are most likely to held responsible for government actions as the largest party in the coalition. So for want of a better reason to do differently I have effectively produced the forecast in the same way as if the Conservatives were in a single party government.  Equally I haven't taken any account of minority governments in the past. Too sporadic to model statistically.  The paper discusses how the coalition affect on Lib Dem and Labour support is factored in because it is already reflected in the polls.

Are past polls a good guide to the future?
  • Interesting data, but why assume that past polls are a good guide?
  • My preference wld be for "fundamentals" forecasts at this stage (economy, maybe PM approval)
I'm not assuming history of polls is definitely a good guide. I'm partly trying to work out how good a guide it is and the confidence intervals reflect estimates of how poor a guide current polls are. There is a meta level question as to whether the past is a good guide to future levels of uncertainty. The out-of-sample prediction section in the paper suggests that it has helped a bit in the past and so it may do again in the future. It may not work out but in trying to say something about the future I think it is better than not considering past experience.

The question of whether there are better bases for forecasting is a good one. My impression is that the "fundamentals" might be good to also consider later on, but at this stage you would be effectively predicting the future trajectory of the economy over quite a long time period; arguably more difficult.

Is this the best forecast we can do?
  • is your view that this model provides the most accurate available forecast, or just that these factors produce it?
I'm not aware of any others this far out apart from assuming that current polls are the best guide we have and projecting them forward assuming no change between now and the election. Part of the point of the paper is to consider the most likely direction and magnitude of change between now and the election. Past experience suggests that the method I'm talking improves prediction by about 30% over just assuming the polls will stay steady. (See paper.)

Regional and constituency factors
  • How's that going to happen without any Tory seats in a city north of Birmingham?
  • Also doubt past precedent is helpful either & believe Tories are losing the North as they lost Scotland
  • You results suggest a Tory lead of less than 10% resulting in comfortable majority. Which seems unlikely
  • boundaries used to favour Tories now favour Labour.
The seats prediction is off the back of 2010 constituency results and so respects the geographical performance of the parties then.  Uniform swing and similar haven't been perfect in the past but they have been sufficiently good so as to think that the seats prediction part is a small problem relative to prediction of the GB share of the vote. Bias in the system is clearly visible in simulations with other results.

Switching between parties
  • Secondly what is going to happen to the 2010 LD>CON switchers. In your model they seem to evaporate.
  • Third point the detail from almost all polls shows that very little of current LAB support comes from CON converts.
  • To what degree does the model take into account the point frequently made by Mike Smithson, that the loss of half the Lib Dem vote from 2010 has mostly gone to Labour, and there appears to be little sign that it will either go back to the Lib Dems, and very little sign that it will migrate to the Conservatives?
Model doesn't look at all at individual level switching, just overall levels of change. The predictions say nothing about who switches which way, just the net effect. Switching up to now is reflected in current poll levels which influence the forecast but just based on current levels of support, not origins or voting histories.  

Taking previous election result into account
  • Historical tendencies? Well you forgot the historical tendency that incumbent parties have only once increased their share of the vote since the 50s.. 40% for the Tories from a 2010 starting point of of 35%? What a crock of shite!
  • You are not taking into account that the GE to Mid-term fall this parliament is less than historical data

The model doesn't directly take the prior election result into account. It should influence things indirectly but affecting the levels of support in the polls in the mid term. I will explore further whether it adds value beyond current polls. What is being suggested in the first comment is actually a model of forecasting without any polls at all. It seems odd to ignore the most current information. I'm just arguing that we should look at with the perspective of how polls in the past have corresponded with future election results.

A long-range forecast for a 2015 British General Election based on current polls and historical polls and votes


I've just started publishing forecast from a polls based method for forecasting the next general election here. I'm hoping to update the forecast weekly. 

As I say on the website: "The methodology is described in a working paper. The approach is broadly to predict the next election based on current opinion polls and the track record of polls in previous electoral cycles allowing for change in opinion in the run up to the election. The method allows for three main phenomena: historical tendencies for the Conservatives to over perform and Labour to under perform their vote intention figures in the polls when it comes to election day; governments being more likely to recover and oppositions fall back in the run up to an election; and a tendency for parties to move back towards their long-run average level of support. All three suggest a Conservative recovery and a Labour set back from autumn 2013. The statistical regression methodology generates estimates of uncertainty and so prediction intervals (range of likely outcomes) and probabilities for key events are also provided below. The forecast represents a way to think about the implications of current opinion polls for the outcome of the next general election in light of the historical relationship between polls and election results. It is the product of a statistical analysis of the data and not my personal opinion about what will happen."

The first forecast is replicated below. There are two striking features. The first is that the Conservatives are forecast to do much better than they are currently polling, mainly because they are in government but also partly because they are historically at a relatively low point (from which regression to the mean effects suggests they should recover) and as a party they have tended to outperform polls at elections. Labour are predicted to do correspondingly worse and so the forecast for the top two is almost symmetrically opposite from the current polls: 40:32 instead of 33:38.

The second of the two most striking features is that the prediction intervals for shares of the vote are enormous.  For the main parties it is clear that there is more uncertainty in the Conservative vote but even the forecast Labour vote could be out by as much as 6.6 points: a huge political difference.  At first glance these prediction intervals may seem to encompass all foreseeable outcomes and more. For the Liberal Democrats the interval from nothing (a hard boundary that had to be invoked!) to 26% seems ridiculously large. While these may seem hilarious at first sight, remember that not all points within the intervals are equally likely to occur. Also as 95% forecast confidence intervals they reflect the historical variation in the votes for these parties and there should be only a 5% chance of a result outside the interval. So they are bound to be very broad to be credible. Even so, the lower bound for the Conservative forecast, at 28%, tells us that it is very unlikely the Conservatives will do much worse than they currently stand in the polls, which is informative. Similarly, the Labour prediction interval suggests it is extremely unlikely that Ed Miliband will do as well as Tony Blair in 1997 or 2001 but it would not be surprising (given polling in previous elections) if he did worse than Gordon Brown or Michael Foot. 

The forecast election-day seats are as you would expect them given the forecast shares. They are a probabilistic estimates of the kind used in election night forecasting. A classic uniform change prediction would produce slightly different figures but not by much, especially given the large prediction intervals for seats that follow from the large prediction intervals for votes.

The estimated probabilities for key outcomes are perhaps the most helpful feature of the forecast so far from an election. These show that the Tories have an 88% chance of being the largest party but only 57% chance of an overall majority. These are rather one sided at this stage in the cycle given there is plenty of scope for change in party support before the election. Even immediately before the election there is considerable uncertainty given the historic record of the polls as shown in Table 1 of the working paper.

Having some estimate for the probability of a hung parliament (currently 28%) is helpful to understand the operation of the electoral system. A Liberal Democrat recovery to May 2010 levels would increase the chances of a hung parliament to over 50%.

Date of forecast: 25.10.2013
Days till the election: 559

Inputted current average poll shares 
Con : 33
Lab : 38
LD  : 11

Forecast Election Day Shares and 95% Prediction Intervals
Con : 40.2 plus or minus 11.8 i.e. between 28 and 52
Lab : 31.8 plus or minus 6.6 i.e. between 25 and 38
LD  : 11.8 plus or minus 14.5 i.e. between 0 and 26

Forecast Election Day Seats
Con : 337
Lab : 265
LD  : 21
Con majority of 24

Forecast Election Day Seat approx. 95% Prediction Intervals
Assuming LD share at 11.8 and allowing Con and Lab to vary as per interval above.
Con between 219 and 471
Lab between 140 and 376
LD between 14 and 30

Probabilities of key outcomes
Pr(Con majority) = 57%
Pr(Lab majority) = 15%
Pr(Hung parliament) = 28%
Pr(Con largest party) = 88%
Pr(Lab largest party) = 12% 

Thursday, 9 May 2013

Local Elections 2013: UKIP hurt Labour at least as much as the Tories compared with last year


Since the local elections last week discussion in the media has focused mainly on the impact of UKIP success on the Conservatives and how they will react.  But compared with last year's local elections Labour suffered at least as much.

By comparison with the last county council elections in 2009 it does look like UKIP gains were at the expense of the Conservatives. Over 80% of UKIP seats came from the Tories, and the Tories lost a greater proportion of their seats to UKIP than the Liberal Democrats, while Labour lost barely any to them. Also there were many seats and councils where the UKIP advance was enough to help another party topple the Tories.

It also makes sense that UKIP should be taking votes from the Conservatives more than others. UKIP are a party of the right. They typically take policy positions that are currently considered to be right wing, such as being against EU membership, immigration, gay marriage and climate change mitigation. So it is unsurprising that UKIP support in the polls comes much more from former Conservative voters than from Labour or the Liberal Democrats

But there are a number of problems with these observations as a way of looking at the results of this year's local elections.  First, UKIP seat gains were very modest because of the even distribution of the vote and effects of the First Past The Post electoral system. Their vote share of 25% where they stood, not the seat tally, is the main mark of their success. Second, the Conservatives did unusually well and Labour especially badly in 2009. The Tories were bound to suffer loses and Labour bound to make some gains, as they did in 2011 and 2012. Third, attitudes on Europe, immigration and moral issues are not highly correlated with economic left-right positions. So there are plenty of eurosceptic, anti-immigrant and socially conservative voters among those with relatively socialist values who might be persuaded to leave Labour for UKIP. Fourth, while the opinion polls tell us about general election vote intention, local elections are a different matter.  UKIP's share of general election vote intention, at 14% last week, is much more muted than the 23% they achieved in the Projected National Share of the vote (PNS) from the local elections.

For these and other reasons, it is more instructive to consider party performance this year by comparison with last year's local elections, rather than those in 2009.  Between 2009 and 2012 we saw a change of government, a recovery for Labour and a collapse of the Liberal Democrats, a decline for the Conservatives, and, significantly, a drop in UKIP support (they were running at over 15% in the polls in the run up to the 2009 European Parliament elections but at just 8% this time last year). There are important political questions about how the trends for the Westminster parties up to last year's local elections have been disrupted by the rise of UKIP since this time last year.

If we look at the opinion polls, while UKIP did seem to benefit from Tory losses in the month after the 2012 'omnishambles' budget, there has been relatively little change between the start of May this year and the same time last year: Con -1, Lab -2, LD no change and UKIP +4.  Given the usual +/- 3% confidence interval, we can only really be sure there has been a rise of UKIP.  If anything it looks like it has come from Labour more than the other two parties.

By contrast, there were dramatic differences in party performance between this year and last. The PNS shows that while all three main parties are down since last year, Labour has dropped more than the other two (Con -6, Lab -9, LD -2). Labour were the biggest losers and this alone suggests that UKIP hurt Labour substantially if not more than the Tories. 

Certainly we can't account for the UKIP rise since last year by reference to Conservative loses alone. Either there was a massive shift of support from Labour to the Tories and also from the Tories to UKIP or considerable direct loses from Labour to UKIP.

The pattern of the results across divisions suggests that relative to last year new UKIP votes have come as much from Labour as from the Tories. As at the general election, the change in the UKIP share of the vote since 2009 is more strongly negatively correlated with the change in the Conservative share than it is with any other party. This is as you would expect if most of the direct switching to UKIP from the Conservatives. However, if we look at the places that had local elections on the same boundaries last year and this year (combining district wards to make county divisions), the UKIP change since 2012 is equally strongly correlated with both the Conservative and Labour change, suggesting that relative to last year both parties suffered equally from UKIP progress. (See footnote.)

The socio-demographic profile of the places where UKIP did well also suggests that it took more new votes from people who would otherwise have voted Labour this year. We are used to seeing the party doing better in places with more routine manual workers who are traditionally part of Labour's base. But the correlation between the UKIP share and the proportion of working class people is stronger this year than it was in 2009 or 2012. Correspondingly, the change in the Labour share since last year is negatively correlated with the percentage of routine manual workers. So it seems that Labour is losing more of its base to UKIP.

Some may argue that Labour can afford to lose some support in its heartlands so long as it does well where it needs to win seats at the next general election. UKIP advanced more in the South and East of England and least in the North West, while there reverse is true for Labour.  This is the opposite of what Labour need to do. 

Moreover, comparison with 2012 is not unfair to Labour. It was not a particularly good set of local elections for them from which they might be expected to fall.  Their lead over the Tories in the PNS then, and more so now, is considerably lower than at any of the local elections while Tony Blair was leader of the opposition. 

One saving grace for Labour is that in divisions where they started second to, but within 20 points of, the Conservatives (i.e. Con-Lab marginals) their vote share was up by 10.5 points since 2009, compared with just 8 points elsewhere.

Otherwise, the distribution of the new UKIP voters this year bodes at least as badly for Labour as for the Conservatives. But as Nigel Farage is fond of pointing out, the main effect of UKIP is probably more psychological than psephological. At the moment it seems that the Conservatives are the most panicked while Labour are rather nonchalant. Whether either reaction is helpful remains to be seen.

***

Thanks to John Curtice and Rob Ford for comments and to them, Jon Mellon and the BBC for help with the data.

Footnote: Admittedly there are only 69 such divisions where the three main parties stood each time and UKIP at least one time and they fought either all or none of the constituent wards in a division in 2012. They are places where Labour are relatively strong, but they are similar to other divisions in their pattern of 2009-13 change and the switch in correlations can be seen within these cases. It is also a phenomenon we have seen before at the 2009 European Parliament elections when the change in the UKIP share of the vote was much more strongly correlated with the Labour change than with that for the Conservatives. Note also that the correlations between Liberal Democrat and UKIP change are just as strong as for the other two parties, but the Lib Dems are reaching the point where it is hard for them to fall much further than they did in 2012. 

Friday, 3 May 2013

Local Elections 2013: Why did UKIP do so well?


With 23% on the PNS, and 147 seats, UKIP did shockingly well in this year's local elections. Why? 

There's lots of good analysis of UKIP voters, especially by Rob Ford.  Factors leading people to vote UKIP include anti-government and anti-main party protest voting, euroscepticism, and anti-immigrant sentiment. There is a lot of public support for the policies that UKIP advocate and emphasise. Polls tell us that there is a clear majority for reducing immigration and close to half the public would vote to leave the EU.

But public concern on these issues has been relatively long standing, and the main reasons for protest voting were there last year as well this year.  True, the economic doldrums have continued for yet another year and faith in the abilities of either the Conservatives or Labour to restore economic health has declined since last year, but not dramatically so. So why did UKIP do so much better this year than in any previous local elections?

One very important reason is that the party fielded more candidates. They contested three quarters of seats this year compared with just a third of seats last year and about a quarter in 2009, when the seats up yesterday were last contested. Looking at the 1550+ BBC keywards, in seats they fought both this year and in 2009, UKIP averaged 27%, up 11 points since 2009. In places they fought for the first time this year, they averaged 24%. So, roughly speaking, we can say that if they had only managed to match their 2009 performance but with 2013 candidature levels they would have averaged 15% instead of the 25% they actually averaged. Roughly 44% of the total UKIP vote tally this year is due to improved performance, but 37% is due to fielding more candidates at 2009 levels of support.  Put another way just under half (45%) of the UKIP increase in overall vote share since 2009 is due to fielding more candidates.

Admittedly this does raise the issue of how UKIP managed to attract so many more candidates. This could be partly due to increased overall popularity, but I'm not best placed to speculate.

So that leaves the question of why UKIP were up 11 points on 2009 in the places they did fight last time. Also in last year's local elections UKIP averaged just 12% in the 311 BBC keywards they contested, 13 points lower than this year's average. This is the same average rise as in the (albeit only 41) places were the county council division boundaries this year are the same as the ward boundaries for last year's elections.  

Polls suggest that UKIP support has risen steadily since the general election from 3% in 2010 to 11.5% now, with a 4% rise since last year.  A 4 point rise in the polls for UKIP provides some explanation for a 13 point increase in local election performance, but still leaves a lot to explain. 

The local election campaign over the last month probably has much to do with it.  UKIP seem to have run a much more effective campaign with many more activists who should help mobilise more voters. But the UKIP share and change in share of the vote is negatively not positively correlated with turnout across the BBC keywards.

Maybe more important received a lot more media coverage than ever before; and the debate over immigration from Bulgaria and Romania, which became the focus of the UKIP campaign, surely helped them.  Protest over wind farms helped mobilise support (but in a limited number of places) but disgruntlement with the planned HS2 route probably didn't make much difference.

What should the main the main parties do in response?  Bonnie Meguid (University of Rochester) tells us that the worst strategy to deal with single-issue or niche parties (like the extreme right and greens) is to try to argue against them. It only makes more people realise they agree with the niche party and not the mainstream parties. As Tim Bale observed after the Eastleigh by-election, by focusing on UKIP issues without addressing them satisfactorily, "rather than shooting UKIP's fox, Cameron has fed it." Better, according the Meguid's analysis, to ignore niche parties or steal their policies (if you can stomach them).

***
Thanks to Rob Ford, Jon Mellon and John Curtice for comments

Why didn't UKIP win so many seats given they got so many votes?


UKIP did extraordinarily well in yesterday's local elections, securing a projected national share of the vote of 23%. At the time of writing they have won over 120 council seats; much more than predicted but relatively few given the number of votes they won.

In the 1550+ BBC keywards Conservatives won 36% of the vote and 53% of the seats.  By comparison, UKIP won 21% of the vote but only 8% of the seats.  

In part this difference is due to a classic First Past The Post electoral system effect, but Labour managed to get 19% of the seats on 20% of the vote, and the LibDems got 14% of the seats on 13% of the vote among the BBC keywards.

The reason UKIP did particularly badly in translating votes to seats is because their vote share is still very evenly distributed by comparison with the three main parties in parliament. The histograms below show how the Conservatives vote shares across different divisions vary much more than UKIP.  More generally, whereas the standard deviation of the share of the vote across divisions for each of the three main parties was 15 percentage points, it was just 8 for UKIP.  This is a pattern that has been true in previous elections.

It seems then that UKIP suffer from what used to be called the 'yellow peril': coming second everywhere and first nowhere.  UKIP came first in 8% of BBC keywards but second 40%, more than any other party. Unless they resolve this problem they will have difficulty winning seats in a general election too. 
 


***
Thanks to BBC, John Curtice, Rob Ford and Jon Mellon for help with the data.