Dodgy use of opinion polls

This graphic has been doing the rounds on Twitter. It is perhaps one of the most misleading use of opinion poll data I’ve ever seen. 

The graphic comes from this Ipsos MORI poll. Here is the link to the charts for the poll and I’ve reproduced the page with the graph on below. 

You can see there are actually two graphs. The one on the left is the one at the start of this blog post and is the one doing the rounds on Twitter. It shows Labour with a 5 point lead. The one on the right is shows a 1 point lead for the Conservatives. 

So what’s the difference? The one on the left gives the voting intention for all the people sampled. The one on the right gives “headline voting intention” which weights the result to take account of the likelihood of people to actually vote. 

This is massively important because if you don’t do this you get a distorted picture. Indeed in the 2015 general election the pollsters overestimated the Labour vote share. 

In their post election analysis the pollsters came to the conclusion that the reason they overestimated the Labour vote share was because they overweighted the likelihood of many Labour voters (particularly younger voters) to vote. 

So by choosing the graph on the left the person who tweeted this has presented a distorted picture that magnifies the errors in the polls done prior to the 2015 general election. No one credible ever quotes opinion poll results without weighting for turnout. 

They have either been guilty of blatant and deliberate cherry picking, or if I’m being generous, an unwitting confirmation bias. Either way it massively distorts the data. 

For the avoidance of doubt this post is not meant to be pro or anti any party, or party leader. It is meant to be pro-clarity and anti-distortion. 

And finally, I will add my six laws of opinion polls to help understand opinion polls more clearly. 

First law: If an opinion poll gives a result you don’t like and your defence is “they can’t be trusted” that’s fair enough.

Second law: If you have ever said “they can’t be trusted” but you have used an opinion poll to back up your preferred issue then you have broken the first law.

Third law: If the poll says what you would like it to say how sure are you that you are not guilty of confirmation bias?

Fourth law: Whilst voting intention polls are not great at precise predictions of vote share, they are broadly right. 

Fifth law: Remember every poll has a margin of error. Find out what it is.

Sixth law: It is dangerous to rely on one poll as the result could be an outlier within the margin of error (see fifth law) and/or you could be guilty of confirmation bias (see third law).

Does buyer’s remorse affect the EU referendum result?

image

I’ve seen a screenshot of the results of a ComRes post EU referendum poll, asking voters if they are happy with the results of the referendum result.

The polls shows that 1% of Leave voters are unhappy with the result. Given 17.4 million people voted Leave this is 174,000 unhappy Leave voters.

There was a winning margin of 1.3 million votes and even if all of the 174,000 Leave voters actually have buyer’s remorse and would have voted Remain, it would still have been a win for Leave, albeit with a reduced winning margin of 0.95 million votes.

For Remain to be able to claim buyer’s remorse is a significant factor they need to be able to show that at least 4% of Leave voters were unhappy.

For the sake of full disclosure: I voted Remain

Edited to add:

This morning a Twitter user @jabd1980 hit me up with the following very pertinet question:

Twitter pic regrexit

So I went onto the ComRes website and found a margin of error calculator. So I stuck in the figures of 33.5 million for the population size (total number who voted) and sample size of 1,069 (given on the screenshot) and got a margin of error of 3.0.

This means the buyer’s remorse effect is just on the edge of the margin of error. But only just.

 

EU referendum result breaks the law

EU ref results as percentage of whole electorateYes, the result does break the law. However, it is not a law that invalidates the result, but it is one rather ironically, that has the word “union” in it.

The EU referendum result gave a 51.9% Leave to 48.1% Remain result on a 72.2% turnout.

The graph here shows that when you look at the whole electorate (not just those voting) you will see only 37.4% of the whole electorate voted Leave and 34.7% voted Remain, with 27.8% of the electorate not bothering to vote.

Why does this matter? Only a few weeks ago the Trade Union Act 2016 came into law and this has impacted upon me and millions of other trade unionists. The government have “modernised” (a euphemism for “massively restricted “) the way strike ballots work as follows (taken from the government website):

The Trade Union Act will ensure industrial action only ever goes ahead when there has been a ballot turnout of at least 50%.

In important public services, including in the health, education, transport, border security and fire sectors, an additional threshold of 40% of support to take industrial action from all eligible members must be met for action to be legal.

As a country we can  take the massive decision to withdraw from the EU based upon 37.4% of the electorate voting for it but cleaners in a school who want to take strike action need to have a 50% turnout and meet an additional threshold of 40% of support from all eligible members to take strike action.

It would not be enough to get a simple majority as with the EU referendum.

So there you have it, the EU referendum results breaks the law: the law relating to trade union strike ballots.

By the way, I am not posting this because I think the referendum result is invalid; I voted Remain but I do accept the result. I am posting this to make the point workers have had a second class democracy imposed upon them. 

Dodgy graphic from West Midlands Combined Authority

image

I was alerted, via Twitter, to the above dodgy graphic from the newly established West Midlands Combined Authority (WMCA).

A quick glance at the graphic and the numbers shows it to be very misleading. Design triumphed over clarity. 

It took me all of five minutes on Excel to knock up the bar chart below. This gives a much more accurate representation of the relative amounts spent. Why couldn’t the WMCA couldn’t do this?

image

Dodgy A&E graph from Vote Leave

IMG_20160522_115443 (1)

Oh my, there really has been some bad maths from the Vote Leave campaign in a report they recently published. The graph above from page 29 of the report make a totally elementary error in its implication that migration caused the increase in A&E attendances.

A phrase every A Level stats student knows (and many who have not studied stats know too) is “correlation does not imply causation.” It is easy to find things that correlate but many of these correlations are spurious. Indeed there is website devoted to such silliness.

Just because things correlate it does not mean one causes the other. For example, as one wag on Twitter demonstrated, the increase in global average temperatures is correlated with the increase in net migration. Does that mean migration causes climate change?

a and e migrtion silly causation

Of course correlation may imply causation but before you make such a bold statement you should consider other possibilities. So, why might A&E admissions be going up? Given I’m a trade union official for UNISON, the biggest union in the NHS, I can speak with a little knowledge from talking to our members. I would theorise that perhaps the fact that we have an ageing population this would mean more A&E admissions.

Is it possible to check this theory? Well yes, it is very easy.

Here you will find A&E data that gives admissions by age group. The age related data only goes back to 2011 (unlike the data in the Vote Leave report which goes back to 2002) but it still provides enough evidence to debunk the Vote Leave claim that migration has caused the increase in A&E admissions.

I compared the data between 2011 and 2014 to see the age related breakdown. I grouped them in three age categories 0-19 (children and youths); 20 – 49 (adults); 50+ (older adults). Remember we know migrants are typically younger and nearly all under 50.

Age A&E admissions

We can clearly see here the biggest increase in A&E admissions are in the 50+ age group. in fact the 50+ age group are just over 50 per cent of the increase in admissions. Given migrants are younger and nearly always well under 50 it is safe to assume that migrants are not responsible for this part of the increase.

Also, this article from the GP website Pulse says:

As many as 5.8 million people attended A&E in 2012/13 after failing to get a GP appointment, representing more than one in four of all attendances, researchers have claimed.

According to the table in the Vote Leave report, net migration was 177,000 in 2012. So these 5.8 million admissions are unlikely to be all down to migrants otherwise it  would mean they’d each visited A&E 30 times in a year!

Remember two facts. First, migrants tend to be younger than the average age of the rest of the adult population. Second, younger people have fewer health problems and use the NHS less than older people.

Of course migrants may have contributed to increased A&E admission but it is simply ludicrous to assume (as Vote Leave have done) that migrants have caused all of the increase when it is clear this ain’t so. And to cap it all after getting a spurious correlation they then use that to extrapolate out as to what the A&E admissions increase will be in 2030. This is a woeful abuse of statistics.

What makes this misrepresentation even more egregious, is that under the graph it says “there is a very strong correlation (>0.9)” and the graph shows the R squared value (or coefficient of determination) to give some technical gravitas to demonstrate “we know what we are doing.” But they don’t know what they are doing as they have ignored the iron rule that “correlation does not imply causation.”

Either they know this and they are being wilfully misleading, or they are shooting from the hip without knowing what they are doing.

If it is the former we cannot really trust anything else they say. If it is the latter, it makes me wonder what else they don’t know what they are talking about.

Pac-Man pie chart of UK’s spend on the EU

good pie

Please note since posting this it has been pointed out to me that this pie chart is not 100 per cent accurate – though the picture it paints is broadly correct. You need to read all the way to the bottom of this post to see info on the inaccuracies.

 

How much of overall UK government spend is on EU membership?

I took the figures from the government’s Public Expenditure Statistical Analysis 2015 and knocked up a little Pac-Man pie chart. I’m not generally a fan of pie charts – see here for why and for info about the inspiration that made me knock up this chart.

Pie charts are generally a bad data visualisation tool. But this one does the job. If you want to see my spreadsheet to check my numbers go here.

If you think the £3.7bn could be used to fix the NHS as some leave supporters claim (they’ve also claimed the savings from leaving the EU could fund other things too) take a look at my bar chart below that shows the £3.7bn would be a small beer in terms of increasing spending on the NHS, education and other public services.

horiz bar

For the sake of full disclosure, I’m a remain supporter.

Edited on 23 May 16 to add:

A few days after I posted this Ian Cuthbert pointed out on Facebook  that I had made a mistake.  He wrote:

I’m interested in the figure for costs of EU membership, as it doesn’t tally with my understanding. Your figure of £3,723m (0.5%) seems to be for EU transactions (in the Public Expenditure Statistical Analysis). But I don’t understand this to be the same as the cost of EU membership, which page 18 of that source shows as just under £8.9bn (Net expenditure transfers ti the EU). See page 18 of Public Expenditure Statistical Analysis – here’s a screen shot of the part showing costs of EU membership. Here’s the link to the document which you included:
https://www.gov.uk/…/public-expenditure-statistical…

In my own fact check on the graph I simply used the ONS and the OBR figure of £8.5 for 2015, but it was an estimate, not a final figure. That figure is, straightforwardly, the UK contribution to the EU, net of the rebate.

So I arrived at a figure of net contribution for EU membership of a little under 1.2%.

In a way, it is a moot point, because none of this includes the net economic benefits of membership. But it is this figure that the Brexit campaign has headlined (constantly!) and also seriously misrepresented. Nor does it include EU spending on UK programmes, such as ESF and ERDF, as far as I can see.

But I’d be interested to know what you think.

I replied:

You ask me what I think. I think:

a) I’ve checked what you’ve said – you are correct. I misread the wrong table.

b) I am happy to be corrected – there is no shame in it. We should all be open to challenge and that’s why I always post a link to my data sources in all graphics I do, along with the link to my spreadsheet with the calculations. Open data and open calculations is exactly where it should be at. Having people check my work and point out mistakes increases my knowledge and it makes me better. No one is always right.

c) Whilst it is of course better to have accurate figures, the point I was making is broadly the same – it is a very small proportion of total government spend. The real point to debate is whether it presents good value, as it is clear even if we stopped paying in it would be relatively small beer in terms of propping up public finances.

d) I will post a correction on my blog [this is it].

e) I’ve done a new pie chart. See below.

And here’s the new pie chart which is broadly the same as the last one

new eu pie

 

How to make a political graph properly

Pie chart of govt spendI appreciate that the title of this blog post may be a hostage to fortune and I may be called out on my attempt to produce a graph properly. But given I’ve called others out on poor graphics I thought I ought to put my money where my mouth is.

Someone posted this graph on my Facebook page and said “no attribution but what do you reckon?”

The original Facebook post of the person who initially posted this chart said “What the UK spends its money on.  If you look very hard, you’ll spot EU membership in there somewhere.”

So, rather than just say it was a badly chosen chart produced in an uneasy to decipher way (which it is), I thought I’d have a go at making this chart better. I should add I may not be using exactly the same data as the original chart as I don’t know where the author got it from (edited to add: see footnote as the original data soruce is now known). But for your info I am using the government’s Public Expenditure Statistical Analysis 2014-15.

So why do I think this was a badly chosen chart? Well, it’s a pie chart and pie charts are nearly always bad charts to use. So what’s wrong with pie charts?

Let’s take a step back and ask ourselves “what is a chart or graph?”

Technically speaking it is the visual display of quantitative information. Incidentally this is the title of a massively influential data visualisation book by Edward Tufte.

Any chart encodes a numerical value visually, for example, as a plot on an x-y graph, as a bar in a bar chart or as a slice in a pie chart.

In a bar chart the numerical value is encoded in the length of the bar.

A pie chart seeks to show the relationship of each slice to the whole, effectively showing the percentage of the whole.  In a pie chart percentage value is encoded in the internal angle of the pie slice.

The issue here is that charts ad graphs are used to compare values and it is far easier to compare by eye the lengths of a bar in a bar chart as oppose to the internal angles of spices in a pie chart.

But sometimes pictures can tell the story better than words. Take a look at the two pictures below.

Bad pie 2

Bad pie 2

The above examples were borrowed from here. There you go:  bar charts are (nearly always) better than pie charts.

So now let’s see how we can improve upon the original pie chart. Note I’ve not used exactly the same data so I done a rough reproduction of the original chart below. As you will see while it is not a busy and hard to read as the original it is still not easy to read and do comparisons across the spend types.
Pie govt spend - ravi
Just for a laugh let’s see it in 3D. You will see adding 3D makes it even more difficult to read accurately as the 3D effect now distorts the way the internal angles are viewed. Never, ever, ever use 3D pie charts. They are the enemy of clarity and understanding.
pie 3d
So let’s turn it into a bar chart instead. See below. You can see it is far easier to compare the spend on each item as all you have to do is quickly compare the bar lengths, which is far easier than comparing pie slices/angles.
bar govt spend
The above bar chart is better, but we can still improve upon it by turning it into horizontal bar chart so the labels are easier to read. And to make it even better, we should order the bars from largest to smallest; add titles; makes the grid lines less dominant; and add a reference to the author and where the data came from.
horiz bar

I once was the one of the view that pie charts should never be used but I’ve softened a little and I think very occasionally pie charts can be used to illustrate data clearly. I’ve knocked up a pie chart to illustrate this point.

Note, just to amuse myself, I could not help myself but to colour it yellow and rotate it to produce a Pacman type chart.

good pie
There you go folks. It is possible to make your point quickly and clearly if you chose the right chart and think about how you present the data.

On the actual issue of EU membership I’m firmly with the remain camp. My reasons for supporting remain are perhaps best summed up by my union’s General Secretary Dave Prentis who recently gave three good reasons for suppporting remain in his blog:

Rights at work – the regulations we rely upon to protect people at work are enshrined in EU law and upheld by the European Court of Justice. Leaving would mean that hard-won rights like paid holiday, fair working hours, equal rights for part-time workers, and maternity and paternity leave would no longer be guaranteed.

Protecting your standard of living – leaving the EU would put working people’s standard of living at risk by creating economic uncertainty. This would risk investment in jobs and damage consumer confidence. It would also make the pound vulnerable, which would push up prices and interest rates.

Protecting public services – we see every day how a weaker economy has meant cuts in public spending affecting everything including the NHS, local services, policing and education. We can’t afford to risk any more cutbacks at a time when our public services are under increasing pressure.

To read his full blog post on why UNISON are campaigning to remain go here.

Edited to add: Since tweeting this post out I’ve been informed that the original pie chart came from this blog and the source data was the Public Expenditure Statistical Analysis of public spending in 2013-14. My source s the same but for the more financial year 2014-15.