Flu deaths reality check

Do thousands of Canadians really die every year from the flu? The flu folks keep saying so. I've already heard it repeated several times this year and flu season has just started. This is what the Public Health Agency of Canada said in a recent press release: "Every year, between 2,000 and 8,000 Canadians die of the flu and its complications."

In a CBC interview a few weeks ago an editor of the Canadian Medical Association Journal said: "Four thousand to 8,000  people die every year of influenza." 

It comes directly from the desk of Canada's chief public health officer. "The flu is serious," he tells us from his website. "Every year, between 2,000 and 8,000 Canadians die of the flu and its complications."

Did you ever wonder how they know that? The fact is, they don't know that. "This is a scientific guess. This is not the truth," Dr. Michael Gardam, director of the infection prevention and control unit at the University Health Network in Toronto and a longtime flu watcher, told me.

The fact is, no one knows how many people die after being infected with the flu virus. The death estimates are not based on body counts, lab tests or autopsies. 

"I think people may have the misconception that every person who dies from the flu is somehow counted somewhere, and they're not," Gardam said.

The "2,000 to 8,000" numbers are based on computer models — a statistical guess that comes out of the end of a mathematical formula that makes a range of assumptions about death and flu.

"They're tossing it into a big computer and they're churning out estimates," Gardam said as he scribbled numbers on a white board to show me how the models work.

One model counts all respiratory and circulatory deaths — that's death from heart and lung failure — as flu deaths.

"As an upper limit, they are looking at everybody who died of a heart and lung problem," Gardam said. "So you could imagine this could include people who died of a heart attack that had nothing to do with flu, but the feeling is that anybody who died of flu should be captured in there, plus a lot of other people."

At the lower end of that model they count the number of deaths officially listed as "influenza" on the death certificate, plus all deaths from pneumonia — even though not all pneumonia is caused by flu.

"That is going to include obviously people who died specifically of those, but it might miss people who died of influenza but who didn't get tested, for example," Gardam said.

Data can include deaths by poison

Another model assumes that every extra death that happens in the winter is a flu death. At the risk of oversimplifying, this is the basic formula of that model: winter deaths (minus) summer deaths = death by flu virus.

That includes winter deaths from slippery sidewalks, snowy roads, freezing temperatures, plus all the winter heart failure, lung failure and deaths from cancer. In the language of the computer model, all excess mortality in winter is considered "death by flu." 

The model extrapolates that the flu virus will cause more deaths across all causes, including "disorders of the nervous system," stroke and "disorders of the digestive system." Which means that according to the model, flu causes 33 more "accidental falls" every year, 18 more "accidental poisonings," and 68 more deaths from "psychotic conditions." But what does flu have to do with deaths from accidental poisonings or accidental falls?  

'If they don't collect that information, how do they know that their policies will work? This is called faith-based medicine.'— Dr. Tom Jefferson

How reliable are the computer model estimates? "I don't think they're reliable at all," Dr. Tom Jefferson told me. He is a Rome-based researcher with the Cochrane Collaboration, and he spends his days reviewing all the research on acute respiratory infections and vaccines. He said hard data on flu deaths "are difficult to get hold of for obvious reasons. So enter modelling, which is nothing more than guesswork, highly sensitive to the assumptions you feed into the model. 'Give me a model and I will make it say whatever you want,' a colleague of mine always repeats."

The models are only as good as the data sets that are fed into them. And death can be complicated. If someone already extremely fragile with heart or lung disease is tipped over the edge with a flu infection, is that a flu death, or a heart death or a lung death? Which database gets to claim it? 

"The only mortality estimates which have any credibility are those based on post mortem examinations and tests which were done before death," Jefferson said.

Flu death statistics not collected

In a perfect world, the flu death statistic would be based on an actual count of confirmed deaths after infection with the flu virus. But that's difficult to do, because autopsies are almost never done, lab tests for the flu virus are rarely done, and someone could die from the complications of flu even though the virus is no longer detectable in their bodies.

The numbers we do have don't even come close to the computer estimates. In Statistics Canada's "deaths and mortality" table, under "cause of death: influenza," there were only about 300 deaths a year between 2000 and 2008. Public health officials don't trust that number. They believe it underestimates the true death toll from flu.

But Jefferson believes the models overstate the risk from influenza. "There are no real figures on deaths from influenza. They don't collect that information," he said. "So if they don't collect that information, how do they know it's a threat? And if they don't collect that information, how do they know that their policies will work? This is called faith-based medicine, not evidence-based medicine."

"Could the deaths be being caused by other pathogens? It's an important question," Dr. Kumanan Wilson told me. He holds the Canada Research Chair in public health policy at the University of Ottawa. He's also a hospital clinician who has seen many flu seasons. 

"We see lots of people coming in with upper respiratory infections and we don't know what causes it. Sometimes if they are really sick, we'll test for influenza. We rarely test for anything else."

One of the few attempts to check the accuracy of the models in assessing flu deaths was done by one of Wilson's master's students, and her thesis is interesting reading.

Using data from three Ottawa hospitals over seven flu seasons, Tiffany Smith did two things. First, she counted the patients who died from flu, according to a doctor's diagnosis. Then, using one of the official flu modelling methods, she ran a computer model to see how close the actual body count matched the statistical estimates. Her result? The statistical model predicted eight times as many deaths from flu as there were actual clinical cases.

"I have found evidence to suggest that point estimates of influenza burden generated using statistical models may not be reliable," she concluded, "and that more research is required to understand the limitations of this methodology."

Remember, that's an unpublished thesis, not a peer-reviewed study. But Wilson said it was a well done paper that posed some important questions.

Flu models versus counts

Getting back to the question of how deadly influenza really is, fate did offer up a chance to check the model predictions when the flu pandemic hit in 2009, and the world faced a new influenza threat called H1N1.

Back then a flu expert told me that the pandemic would be a rare opportunity to check the true death toll from flu, because, for the first time, there was widespread lab testing, a national reporting system, and all eyes were on potential flu-related deaths. The final count: 428 deaths, which is much closer to the seasonal average of around 300 recorded in the vital statistics tables than to the 2,000 to 8,000 deaths estimated for the average flu season by the computer models.

So how did the models rate after a real life test? "The predictive models of 2009 of influenza have actually been a complete failure," respiratory-infection expert Jefferson said.

"Ranges like 2,000 to 4,000 or even 8,000 influenza-related deaths a year are thrown around each flu season, and policy decisions and flu shot campaigns are based on these numbers," Michael Gardam told me. "I think it is important for us to remember that these numbers are estimates and certainly not written in stone. These numbers vary a great deal depending on which research paper you read."

There's another point to consider here. Using death estimates is the scariest way to talk about the risk from flu, because 8,000 sounds like a lot of deaths. But if you ask, "8,000 deaths out of how many people?" suddenly the risk seems much smaller. In fact, it would be 8,000 deaths among 35 million Canadians. In other words, in a normal flu season, about  0.02 per cent of Canadians are in danger of dying from the flu, using the highest estimate. Another way to look at it is this: 99.98 per cent of Canadians will not die of flu this year.

Undermining flu campaigns

So are the statistical models exaggerating the death toll from flu? "Not enough people have been asking these questions,"  the University of Ottawa's Wilson said. "These are complicated models. There are multiple ways to calculate the information. Five different analysts with the same data can come up with five different estimates. It depends on how they calculate base line risk, how they define when the season begins, how to run the model. There are lots of potential variables in the model that will influence your answer."

For proof of how models keep changing their estimates, look back at Canada's flu files. More than a decade ago, flu was estimated to kill about 500 to 1,500 Canadians every year. But in 2003 Health Canada changed models, and the estimates jumped to "700 to 2,500 per annum." The 2,500 deaths at the upper end of that range quickly became the lower end, when an even newer model was tried in 2007, pushing the upper limit to 8,000 based on the severe flu seasons of 1997 to 1999.

"Influenza prevention has become an industry fuelled by poor science and propelled by conflicted decision makers," Jefferson said. "This is the significance of the upward creep that you have been witnessing and the chasm that now exists between policy makers and evidence.

"The proof of what I am saying is in the answer to the question: How many die every year? Answer: maybe 300 or maybe 9,000. We are not sure. If you do not know, how can you have such a costly policy and most of all how can you evaluate it?"

When I asked him if there are consequences from over-stating the mortality impact of flu, Jefferson answered: "Yes. Scaring people justifies evidence-free policies. Yes, no one knows exactly what the threat is. The only certainty are the returns for industry."

Wilson is concerned that overstating deaths could undermine the annual flu campaign. "I think this is a potential risk," he said. "It's a good idea to try to capture the number of deaths. People just need to reflect the fact that there is a lot of uncertainty in these numbers and that has not necessarily been conveyed. Even if the estimate is 1,000 or 2,000, it's a big number. A more conservative approach might be better to convince people it's a real disease that we have to take seriously."

One expert I talked to suggests that at least some of the cost of the annual flu campaign should be directed at finding out how much death the virus actually causes every year, by using a system of doctors and hospitals to track laboratory confirmation of flu infections and flu mortality. 

The flu virus has lots of ugly company in the winter — less famous viruses such as RSV (respiratory syncytial virus); the ubiquitous cold bugs, including the coronavirus and the adenovirus; as well as Streptococcus pneumonia and all of its bacterial friends. Influenza is certainly one of the nastiest viruses in the group. It also happens to be the only one with a vaccine.

"You've got to wonder: The stuff we're attributing to influenza, how much of that is actually true and how much of that is other viruses? We don't know because they haven't been studied," University Health Network's Gardam said.

Just 1 death this year

For the record, how many official deaths from flu have been reported so far this year? One.

And finally, as promised, here's the official response I received  from the Public Health Agency of Canada:

Q1. How are the numbers derived? (i.e., how is it counted? are there any statistical models?) The number of flu related hospitalizations and deaths is not a straightforward estimate, given that influenza is such a non-specific illness and its diagnosis is under-reported. Patients with influenza complications or an exacerbation of their underlying chronic medical condition are often not reported as influenza related.

PHAC has taken data collected by Statistics Canada and hospital discharge records from the Canadian Institute of Health Information and applied statistical techniques to provide an estimate of influenza related deaths.

Q2. Are the numbers an average over the last 10 years? Have the numbers stabilized?

As previously indicated, it is difficult to assess the true burden of influenza in terms of incidence, deaths and hospitalization. However, it is estimated that, on average, the flu and its complications send about 20,000 Canadians to hospital every year, and between 2,000 and 8,000 Canadians die.

Q3. Why is it important to inform Canadians about  these death statistics?

Reporting on these death statistics informs Canadians that infection with influenza can be severe and in some cases result in death. Hence, Canadians should get their seasonal flu shot to prevent infection and to practice infection control measures such as hand washing, cough etiquette and staying home when sick to prevent spread.