Should you trust polls when it comes to Toronto's mayoral byelection? It's complicated

It's a mayoral race unlike any we've seen in recent memory. 

And the byelection to fill Toronto's top job has pollsters in the field trying to map out public opinion on a weekly basis. Over two dozen public opinion polls have been conducted since former mayor John Tory resigned in February, all gauging where the large field of potential replacements stands in the eyes of voters.

So, should you trust their findings? The answer is complicated. 

Robert Hutton, Mission Research's director of insights and the former chief operating officer of Pollara Strategic Insights, said voters should approach all public polls with some skepticism at this point. He has concerns about the method used to gather many of the polls popping up in the race.

Hutton said the polls have the potential to miss some key voter groups. But he's also cautioning against ignoring them altogether, he said.

Those polls, conducted by interactive voice response (IVR), are essentially robo-calls. A computer dials a number with a pre-recorded survey and gathers responses. The trouble is people tend to ignore robo-calls or they're simply filtered out as spam, Hutton said.

"The problem with that is that there's only a certain segment of the population that don't screen out those calls," he said. "It misses a whole lot of people. And if those people that it misses have certain common characteristics, perhaps common voting characteristics or attitudes, it'll misrepresent those people in the pool."

So, why do pollsters use IVR? Hutton said it's cost-effective and can be deployed relatively quickly.

Robo-call method still useful, with caution

But it's not the only way polling companies gather their data. Many research firms use a mix of landline and cell calls with live operators and online polling to conduct surveys. Those methods take a bit more time and can cost a minimum of $20,000. But Hutton says they allow for stronger results and a more representative and comprehensive sample.

Despite that, he's not eager to throw IVR out the window.

"We should look at it and say, is there a legitimate trend here?" he said. IVR polls in this race have consistently found Olivia Chow is the frontrunner, Hutton says, which he believes is accurate because that's been the finding made by all of the pollsters.

But he stops short of drawing any conclusions about the pack behind her, noting field placement has varied week-to-week, depending on the poll.

"Now, does that tell me that we have great accuracy on IVR polls further down the list? In the second, third, fourth, or fifth place candidates? My suggestion is probably not."

John Tabone, the Chief Executive Officer of the Canadian Research Insights Council (CRIC), the industry association for research agencies in Canada, said there's nothing wrong with IVR polls, but people should demand transparency from the pollster. That means being able to have a clear look at the sample taken, the demographic breakdown of their numbers and the unweighted sample.

An unweighted sample is the total number of people asked the survey questions but includes those that weren't factored into the poll results.

Those figures should give people an idea of how the company arrived at its findings and if it roughly represents the census data of the place they're polling. That can be done with IVR, he said.

"If someone is doing a survey using IVR, they should be providing detailed tables with weighted and unweighted columns," he said. "If they're not providing that, I would be incredibly suspicious and certainly ask them for it."

Take early polls with 'grain of salt'

Tabone said members of CRIC must disclose that data and also if they're polling on behalf of a candidate in an election. The goal is to be as transparent as possible about the process, he said.

"If there's very limited information about how (a company) is getting this data, that's a big alarm bell," he said.

Tabone said that the Toronto race will be fascinating to watch in terms of how the polls tracks ahead of the vote. With such a large field of contenders and weeks to go in the campaign, the race could shift, he said.

"There's probably going to be a lot of movement," Tabone said. "I think people are still just getting familiar with the candidates."

Hutton said these early polls should be taken with a "grain of salt." They also shouldn't discourage voters, he said. Every ballot cast in this race will be important on June 26.

"A poll is always a rearview mirror, it's always about what happened yesterday or two days ago," he said. "It would be very premature to think of any of those polls as being conclusive and an indication of how the election is going to turn out."