The popular adage that the only poll that matters is the one held on election day took a knock last week when London markets were sent into a tailspin by a survey of a mere 1000 respondents in Scotland.
The occasion was a finding by the prestige market research firm YouGov that Scottish voters were about to put three centuries of British union to the torch at Thursday’s independence referendum. While the poll was far from the first to have detected the stunning surge in support for independence that has undoubtedly unfolded over the last month, it was the first to find the “yes” vote had actually tipped over the edge, barring one highly dubious outlier result in August last year.
But with the 51%-49% result landing well within the poll’s margin of error, the poll in fact offered no basis for saying any more than that the situation was very finely balanced. Nor has the notion of a lead for “yes” found support in the seven polls that have emerged since, apart from one poll with a conspicuously smaller sample size than the others.
Among these were a new finding from YouGov published on Thursday, this time showing showing “no” leading 52%-48%. This led Britain’s newspapers, most of which have been vocal in their opposition to independence, to proclaim that the momentum to “yes” had reversed — in some cases with an an unmistakable tone of enthusiasm.
As is so often the case, reportage at both ends of YouGov’s oscillation has been infected by over-analysis of what is probably statistical noise. This point is illustrated by the trend chart of all published poll results below, suggesting that the “no” vote has had its nose in front at all times — although the trend to “yes” might indeed have levelled off at a critical moment, leaving “no” with a lead of about 52-48.
However, poll aggregates are not without error margins of their own, and the present margin is certainly close enough to be within the zone of statistical uncertainty. That’s even before taking into account the particularly strong possibility that the polling is afflicted by a systemic bias, the direction of which will not become apparent until the votes are counted.
A peculiarity of the Scottish referendum polling is that it has largely been conducted online, making for an unhappy contrast with the mix of approaches typical at national elections. Unlike phone and face-to-face polling, online methodologies do not presume to capture random samples of the population, as the universe of respondents is limited to volunteer panels of generally around 50,000 participants. The four most prolific pollsters over the past few weeks have all been employing this approach — and as one of them conceded last week, it is likely that these panels have many of the same people on them.
Owing to the non-random nature of their sampling, the online pollsters are heavily reliant on the veracity of their demographic models of the electorate. In the case of regular national elections, these models can be refined through decades of trial and error. However, the unprecedented circumstance of the independence referendum allows for no past experience to be drawn on. This is particularly vexing given that the voting age for the referendum has been lowered to 16, and polling shows around three-quarters of non-voters from the last Scottish parliamentary election say they will definitely vote at the referendum.
All of which provides fertile ground for arguments on either side that polling is understating their true strength. Optimists in the nationalist camp argue that the online panels are lacking the traditional non-voters who were flocking to voting registration stations in their enthusiasm to vote “yes”.
Conversely, unionists are hopeful that tight polling will cause waverers who might have been willing to lend moral support to a losing cause to step back from the precipice. There is also talk that polling may be blighted by a “spiral of silence”, in which “no” supporters feel reluctant to own up to the fact because their cause is perceived as unpatriotic.
It would seem to be the latter arguments that have the most meat on their bones. For one thing, polling indicates that traditional non-voters, in whom the “yes” camp puts so much store, are actually coming down disproportionately in favour of “no”. For another, the historic tendency for constitutional referenda, in Britain and elsewhere, has been for polling to overstate support for the change option. Furthermore, the two polls that have emerged over the past week that didn’t rely on online methodology both supported the case that “no” has its nose in front — by 54%-46% in one case, and 51%-49% in the other.
For all of these reasons, the money — smart or otherwise — remains with “no”, with bookmakers’ odds generally implying an 80% probability that Scotland will vote to stay in the union.