Here's what pollsters missed on Quebec's vote — and what those lapses mean for the 2019 federal election

Quebec's historic election presented a challenge to pollsters. It wasn't until after the results showed that the polls had largely fallen short that the extent of that challenge became clear — explaining in part why the polls missed the mark and providing some lessons that pollsters will take into the 2019 federal election.

Going into election day, the Coalition Avenir Québec looked to be in a decent position to form a majority government. It had a decisive lead among francophones and had pulled a few points away from the Liberals. 

But it was still a close race. The CBC's Quebec Poll Tracker, an aggregation of all publicly available polls, put the CAQ at just under 33 per cent, and the Liberals at just over 29 per cent. The widest gap between the CAQ and the Liberals in any poll was five points, but most had it at two or three points.

When all the votes were counted, however, the CAQ received just over 37 per cent of ballots cast. The Liberals took a little under 25 per cent. That represented an error of some nine points between the two leading parties compared to the margin the polls suggested separated them.

That is a large error — the biggest in any provincial or federal election in Canada since 2013 in B.C. That was five years and 14 elections ago, when the New Democrats averaged a lead of about seven points in the polls before losing to the Liberals by a margin of four points.

Results from Quebec left a few pollsters scratching their heads and raised questions regarding the reliability of polls, particularly with the federal election only a year away. Pollsters will have little time and few opportunities to tweak and test their methodologies — but already the reasons for the miss in Quebec are coming into focus.

When who votes makes all the difference

After pollsters gather their sample of respondents, they weight that sample to make sure it resembles the general population. In Quebec, one of the most important weighting factors is language — getting the right mix of francophones to non-francophones is essential, particularly since voting patterns between these two groups are vastly different.

But if the general population does not resemble the voting population, that weighting scheme can cause errors. Several pollsters contacted by CBC News identified this as one of the contributing factors to the miss. The numbers back them up.

Turnout province-wide in the election was 66.5 per cent, but in seats won by the Liberals turnout averaged just 59.3 per cent. If that pattern was repeated across the province, it means that many Quebecers who voted for the Liberals in the past — and might have told pollsters they were intending to vote Liberal on election day — did not bother to cast a ballot.

This was especially the case in ridings with significant non-francophone populations. Eight of the nine ridings with the highest population of non-French speakers had turnout below 60 per cent. A post-election survey by Léger found that 47 per cent of non-francophones did not vote.

But if the vote was depressed in safe Liberal ridings, reducing the party's overall popular vote, it was enthusiastic in ridings won by the CAQ, increasing their province-wide score. Turnout in ridings won by the CAQ averaged 69.4 per cent, 10 points higher than in Liberal ridings.

Liberal supporters staying home and CAQ voters turning out in big numbers explains part of the polling miss. The voting population was less Liberal and more caquiste than the population as a whole.

A volatile electorate finally makes up its mind

Turnout alone, however, cannot explain the entirety of the errors. The second culprit identified by pollsters relates to the volatility of the electorate — a unique feature of the 2018 campaign.

Quito Maggi of Mainstreet Research said that in the last four days of their polling, the proportion of Liberal voters saying they were "very likely" to change their minds doubled to 11 per cent. Maggi said volatility was high during the campaign, as their day-to-day tracking sometimes picked up double-digit swings in support.

According to Jean-Marc Léger, writing in Le Journal de Montréal, "our post-election poll shows that 36 per cent of voters made their final decision in the last week, including 11 per cent in the polling booths. It was eight per cent in 2014!"

About five per cent of Liberal voters crossed over to the CAQ in the final weekend, according to Léger. He said 54 per cent of Quebecers considered supporting a party other than the one they finally voted for, a degree of volatility he had never seen before.

"I don't know what can be done to catch a last-minute swing like this," said Joseph Angolano of Mainstreet.

Lessons for the 2019 federal election

Mario Canseco of Research Co. thinks one solution is to be in the field as late as possible. "It's not about predicting the result a couple of days ahead," he said. "It's about getting the freshest possible look at the electorate."

"But this was an atypical election," says Sébastian Dallaire of Ipsos, "a reversal of the last 50 years of politics in Quebec. With a referendum on independence off the table, Quebecers were looking for new alternatives."

These kinds of elections can be more unpredictable, but that might not be the case federally in 2019.

"The federal election is unlikely to be so outside of the norm," says Dallaire, "as there is no strong desire for change or a long-sitting government. It is a more typical election cycle."

Differences in turnout may also not be as big of a factor in the federal campaign, as there isn't the same kind of regional or demographic voting block that could have a similar impact nationwide as the disparity in turnout between francophones and non-francophones had in Quebec.

Still, Dallaire thinks better questions can be asked to provide a greater understanding of who is likely to vote and why voters are supporting one party over another.

Several pollsters expressed reservations about implementing likely voter models, as is used more widely in other countries. There is a poor track record of these models working in Canada. After the debacle in 2013 in B.C., pollsters tried to model turnout for the 2014 Ontario election. In every case, these models performed worse than simply weighting to the general population.

But Dallaire said that more information about what is motivating voters could provide more context about potential outcomes. Angolano agrees.

"Perhaps we could signal in our projections which party has the best chance to over or under-perform on our final estimate. We'll just keep working to improve. This is science after all." 

Emphasizing the known unknowns

Dallaire believes that how we talk about polls might need to change, with more emphasis on the potential for volatility in an age when fewer Canadians identify with any single party.

He also hopes the results in Quebec begin a discussion about the role of public opinion research.

"It is important for people to know what their fellow citizens think," said Dallaire. "The only countries that ban polls are authoritarian dictatorships. But it's also important to be transparent and open about how we are conducting polls, so that people can have confidence in the results."