One of the questions that often gets asked is whether a given pollster generally delivers a higher vote estimate for a party than other pollsters — basically, whether a polling firm such as, say, Newspoll (to choose a random polling organisation), leans towards one party or the other.

We can never really tell if any pollster delivers results that are actually higher or lower for a party than other pollsters, because we just don’t have elections every week to determine the true state of public opinion with which to judge them against. However, we can look at relative leans — how pollsters lean for or against a party on the vote estimates compared to what other pollsters are doing at the same time.

It doesn’t tell us who is more accurate — and that’s an important factoid to keep in your thought orbit here — but rather, it tells us how pollsters behave comparatively to each other.

To get us into the groove — and something you may not have seen in a while — this is how the primary vote estimates and the two party preferred vote estimates look like since September 2010  for the four public pollsters we regularly track.

These charts are interesting enough — you can sort of see the way some polling firms look like they produce results often more favourable to one party than the other. However, to really examine any relative lean, we need to go a little deeper.

The first thing we need to do is have a yardstick from which to compare the pollsters against. Thankfully, we already have a perfect tool for this — our Pollytrend estimates. Just to refresh, our Pollytrend estimates are based on an aggregation of the most recent poll from all pollsters we track, weighted by both sample size and time. So the older a poll is, the less weight it has in our trend and similarly, the smaller the sample size, the less weight it has in our trend. As a new poll gets released by a pollster, that new poll replaces the previous poll of that pollster in the algorithm. As far as I can see, there isn’t a more theoretically accurate estimate of the true state of political public opinion in Australia than our Pollytrend series — which makes it kind of handy for what we want to do.

The other thing we need to be mindful of here is to only compare temporal like-with-like in the polling results. Not all pollsters produce the same amount of polls, so we have to take that into consideration. Essential Report comes out every week, Newspoll once a fortnight, Nielsen once a month and Morgan’s Phone Poll (we don’t use its face-to-face results here) gets produced whenever it gets produced.

To control for the different quantities of polls for each of the pollsters, we’ll compare its poll results to the Pollytrend result that occurred on the last date that a given poll was in the field. So each pollster gets each of their polls compared to the Pollytrend result that existed at the time each poll was undertaken. Rather than do it for the primary votes and the two party preferred, we’ll just use the two party preferred results — and we’ll use ALP two party preferred results as our reference.