One of the questions that often gets asked is whether a given pollster generally delivers a higher vote estimate for a party than other pollsters – basically, whether a polling firm such as, say, Newspoll (to choose a random polling organisation), leans towards one party or the other.
We can never really tell if any pollster delivers results that are actually higher or lower for a party than other pollsters, because we just don’t have elections every week to determine the true state of public opinion with which to judge them against. However, we can look at relative leans – how pollsters lean for or against a party on the vote estimates compared to what other pollsters are doing at the same time.
It doesn’t tell us who is more accurate – and that’s an important factoid to keep in your thought orbit here – but rather, it tells us how pollsters behave comparatively to each other.
To get us into the groove – and something you may not have seen in a while – this is how the primary vote estimates and the two party preferred vote estimates look like since September 2010 for the four public pollsters we regularly track. Click to expand each chart
Get Crikey FREE to your inbox every weekday morning with the Crikey Worm.
These charts are interesting enough – you can sort of see the way some polling firms look like they produce results often more favourable to one party than the other. However, to really examine any relative lean, we need to go a little deeper.
The first thing we need to do is have a yardstick from which to compare the pollsters against. Thankfully, we already have a perfect tool for this – our Pollytrend estimates. Just to refresh, our Pollytrend estimates are based on an aggregation of the most recent poll from all pollsters we track, weighted by both sample size and time. So the older a poll is, the less weight it has in our trend and similarly, the smaller the sample size, the less weight it has in our trend. As a new poll gets released by a pollster, that new poll replaces the previous poll of that pollster in the algorithm. As far as I can see, there isn’t a more theoretically accurate estimate of the true state of political public opinion in Australia than our Pollytrend series – which makes it kind of handy for what we want to do.
The other thing we need to be mindful of here is to only compare temporal like-with-like in the polling results. Not all pollsters produce the same amount of polls, so we have to take that into consideration. Essential Report comes out every week, Newspoll once a fortnight, Nielsen once a month and Morgan’s Phone Poll (we don’t use their face to face results here) gets produced whenever it gets produced.
To control for the different quantities of polls for each of the pollsters, we’ll compare their poll results to the Pollytrend result that occurred on the last date that a given poll was in the field. So each pollster gets each of their polls compared to the Pollytrend result that existed at the time each poll was undertaken. Rather than do it for the primary votes and the two party preferred, we’ll just use the two party preferred results – and we’ll use ALP two party preferred results as our reference.
Once we separate the pollsters and look at how their results compared with the Pollytrend results occurring at the same time, this is what it all looks like. Just click on each chart to expand.
You can start to get a feel for the way each pollster leans relative to what the pollsters were saying collectively. To make it more interesting, we can take the difference between each pollster’s ALP two party preferred result and the equivalent Pollytrend – again, click to expand the charts
Taking the Essential Media chart– the producer of Essential Report – to use as an example, what we see is that after March this year, Essential Report consistently produced ALP Two Party Preferred results that were a point or two higher than our Pollytrend. At the other end, Nielsen up until July this year produced ALP two party preferred results that were consistently a few points lower for Labor than what our trend measures were showing at the time.
If we average these differences out, we find that two pollsters lean towards Labor (in that they produce results usually more favourable for Labor compared to Pollytrend) and two lean away from Labor (producing results generally more favourable for the Coalition)…. which makes sense considering.
All our pollsters here have relative leans under 1%, so it’s hardly earthshaking stuff – and it certainly isn’t “bias” in any respectable sense of the word. Rather, Nielsen and Morgan tend to be more favourable to the Coalition by a small margin while EMC and Newspoll tend to be more favourable to Labor by a small margin – at least compared to what the aggregated results of all the pollsters together were saying at any given time.
So yes – our pollsters do lean relative to each other, but not by much, and at varying levels of consistency.