Calgary's civic election polls were a 'comprehensive failure,' report says
Robson Fletcher | CBC News | Posted: August 8, 2018 10:30 AM | Last Updated: August 8, 2018
Authors see plenty of blame to go around but cite failures by Mainstreet Research and Postmedia, in particular
A new report into confusing and conflicting polls ahead of last year's civic election in Calgary says there's plenty of blame to go around but singles out one polling firm and one news organization, in particular.
The report says polling done by Mainstreet Research for Postmedia — the company that owns the Calgary Herald and Calgary Sun — was especially off the mark and detrimental to public trust in the polling industry.
"Their polls were seriously, methodologically flawed," said Paul Adams, a co-author of the report along with Christopher Adams and David Zussman.
But the failures weren't limited to just these companies, according to report, which found broader problems with other polls and other news reporting, as well.
It all added up to a "comprehensive failure from the perspective of Calgary voters," Adams said.
The municipal election campaign was initially expected to be a relatively low-key affair but was then turned on its head last fall, the report says, when the Calgary Herald and Calgary Sun first published the results of a Mainstreet poll that surprised virtually everyone who follows municipal politics in the city.
The poll suggested incumbent Mayor Naheed Nenshi was trailing upstart challenger Bill Smith, who had little public profile at the time, by nine percentage points. Two subsequent Mainstreet polls put Smith even further ahead: by 17 and 13 points, respectively.
The report's authors — two of whom have a background in polling themselves, and the third in government and public policy — found the polls "significantly affected the course of the campaign."
"They threw Nenshi's campaign on the defensive, gave impetus to Smith's campaign, and possibly doomed the prospects of another candidate, Andre Chabot, who Mainstreet's poll suggested was not a close contender," the report reads.
Nenshi went on to win the election by eight percentage points.
Mainstreet president Quito Maggi later admitted to "big, big polling failures," despite frequently and aggressively defending his polls from a variety of criticisms during the campaign.
The report contrasted the humility expressed after the election with public comments from Mainstreet officials during the campaign.
"In the media and on social media, they expressed a kind of bloated self-confidence that was not justified by what they knew to be problems in their methodology," the report reads.
Where the polls went wrong
Maggi, who voluntarily participated in the external investigation, said Postmedia requested a conference call with him after the release of the first poll during the campaign to discuss the public criticism it was receiving.
He said he then decided to change the methodology for the second poll by increasing the sample size and ensuring there was a minimum number of people surveyed from each of the city's 14 electoral districts.
But the report found that only made things worse.
That's because instead of using random-digit dialling to reach people by phone, Mainstreet used a "directory" that it had procured from a variety of sources. The firm then relied on postal code data from that directory in order to determine which electoral district a survey respondent lived in.
Any respondents who had no postal code associated with their phone numbers in the directory were excluded, and that often meant people who only had cellphones were cut out of the results.
"They ended up with an over-representation of landlines with this new methodology," said Christopher Adams.
"And so, with very low cellphones in that sample, they had a distorted youth/adult sample at the end."
The report also found methodological flaws with other polls released during the campaign, including a survey commissioned by a pro-transit group called LRT On The Green. That poll suggested Nenshi was up by 15 points up over Smith but the report found it asked leading questions.
Another survey commissioned by a group of academics and conducted by Forum Research was the most methodologically sound, according to the report. That poll put Nenshi up by 17 points over Smith but it was not intended as a strict horse-race poll and was conducted over a period of weeks, rather than days, making it less useful as a "snapshot in time" of voter intentions.
Overall, the report acknowledges that polling has become significantly more difficult compared to the recent past, when landlines were ubiquitous and random-digit-dialling was standard. It recommends a higher degree of transparency from pollsters when it comes to their increasingly "novel" methods of reaching respondents and calculating results.
The report also recommends news organizations develop a greater commitment to high-quality reporting on polls, while detailing how many failed to adequately inform voters during last year's election campaign.
Where the media went wrong
The city's newsrooms had shrunk significantly since the previous municipal election, the report found, leaving fewer and less experienced reporters to cover the 2017 campaign, resulting in coverage that was less sophisticated.
Some journalists tried to be balanced in their coverage but "were constrained by an uncertain grasp of the methodological issues," the report says, a problem that was further exacerbated by a lack of transparency from pollsters.
"It should be said that Postmedia was in a better position than other news organizations to evaluate Mainstreet Research's public claims of absolute confidence in its work," the report says.
"As reported to us by Maggi, Postmedia executives requested a conference call to address the criticisms of the first poll. In the course of that conversation, Mainstreet indicated it would be tweaking its methodology ... Despite Postmedia's awareness that Mainstreet Research was concerned enough to make changes to its polling, the Calgary Herald and the Calgary Sun continued to print quotations from Mainstreet executives that proclaimed a pugnacious self-confidence about their methodology."
The newspaper company did not participate in the external investigation, telling the report's authors that "legal action citing Postmedia in a separate matter will prevent us from being able to take part in this review."
Lorne Motley, editor-in-chief of both the Calgary Sun and Calgary Herald, declined to comment on the release of the report, citing the same, unspecified legal matter.
Maggi said Mainstreet's contract with Postmedia expired in February and has not been renewed.
Mainstreet welcomes release of report
For his part, Maggi welcomed the release of the external report, saying it came to many of the same conclusions as Mainstreet, itself.
"We really wanted this report to published," he said. "The report is extremely detailed and looks at the errors in all the polls."
In line with the report's recommendations, he said Mainstreet has already made changes to the way it reports its polls, in order to be more transparent about its methods and results.
The company had previously concluded a pair of its own investigations, one looking into its polling methodology and the other into the way it communicated with the public during the election campaign.
Those internal investigations were separate from the results of the external investigation that were released today, which was initially commissioned by the Marketing Research and Intelligence Association (MRIA).
The MRIA is a voluntary, self-regulatory body that represents the market and survey research industry in Canada, but it has run into financial difficulties recently. Last week, it announced plans to disband.
The MRIA had previously received the report but had yet to release it. As a result of the news of the MRIA's demise, the authors opted to release the report, themselves.
Maggi also said he was pleased the report found no evidence Mainstreet was "in cahoots" with any politician or news organization to deliberately skew poll results, as had been suggested by some members of Nenshi's campaign team.
"You couldn't pay me to tank a poll," he said. "There's not enough money in the world."
You can read the full report below.
- MORE CALGARY NEWS | Swerve magazine shuts its doors after 14 years of featuring ordinary Calgarians and their extraordinary lives
- MORE CALGARY NEWS | Students allege inhumane treatment of lab animals at U of C's psychology department
Read more articles by CBC Calgary, like us on Facebook for updates and subscribe to our CBC Calgary newsletter for the day's news at a glance