Lower response rates threaten census data in some places

The response rate to Statistics Canada's replacement for the cancelled long-form census varies wildly from community to community, information released Monday shows.

New data on the agency's website about the contentious national household survey show that the final response rate across the country was 68.6 per cent — down slightly from an estimate made public earlier this month.

But almost 12 per cent of communities had response rates that fall below the optimal 50 per cent level.

Most of those communities with low response rates are small, prompting questions about how reliable the final results will be at a local level.

"The reason we take the census is not to get the data for the whole of Toronto. The point is to get small-area data," said Ivan Fellegi, the chief statistician until 2008.

"My whole point was, and still is, that some data will be good, some will be bad. We won't know which is which."

A solid response rate across the board is essential for gleaning unbiased conclusions from the national household survey. The survey was conducted earlier this year as a replacement for the long-form census that was canned by the federal Conservative government in 2010.

The national household survey is voluntary, unlike the previous long-form questionnaire, which was mandatory. The questions on both forms are the same, but analysts and even Statistics Canada have voiced concern about whether vulnerable groups would be properly represented in the results, since some groups tend not to respond to voluntary surveys.

Updated response rates posted this week

In the data posted on Monday, Statscan gives response rates for the first time for the county, provinces and census subdivisions — what most people call municipalities. Statscan gives the raw response rate, and a second response rate that is weighted to take into account a second round of questioning of non-responders conducted by the data-collectors.

The data show that the large urban centres all had relatively high response rates. In census subdivisions with populations over 200,000, the weighted response rates were between 73.5 per cent and 82 per cent. That's above the national average of 68 per cent and well above the 50-per-cent bar set out by Statistics Canada as optimal.

Toronto, for example, had a weighted response rate of 77.6 per cent.

In communities with populations between 4,700 and 5,300, weighted response rates varied between 57 per cent and 89 per cent.

But in 572 communities out of the 4,949 that were sampled, response rates fell below 50 per cent. Those communities are mainly quite small.

Tilley, Alta., had a weighted response rate of just 37.2 per cent, for instance.

Statisticians expected to see fluctuations in smaller areas and have some strategies to deal with the wide range of results, said Marc Hamel, the census manager at Statistics Canada.

Data quality concerns for some communities

Some of the low response rates have already been dealt with by the weighting process, he said. And if the low response rates are in areas where the 2006 census suggested there is homogeneity, there is little concern about the 2011 data.

As for the areas where low response rates mean analysts can't draw reliable conclusions, Statscan will likely aggregate them with neighbouring areas to make for fuller data, Hamel said.

At this point, the agency's officials are still combing through data and it's too early to say whether some of the local data will have to be left unpublished, he added.

"There's a lot more work to be done."

None of this was comforting to Fellegi.

"There is a huge range," he said after looking at the spreadsheets posted by Statscan. "That says that in some areas, the results will be reasonably good. And in others, they aren't. That was my point right from the beginning."

Even if big cities have high response rates, the swings in the data within smaller communities suggest to him that there may well be similar swings within certain groups in urban centres. Good data for the City of Toronto as a whole says little about how the Chinese population is faring, or whether low-income groups in the downtown core have enough daycare facilities.

"We don't have response rates for vulnerable groups," he said.

Before it had the 2011 data in hand, Statscan ran simulations of the national household survey for three cities — Toronto, Winnipeg and Bathurst, N.B. The tests pointed to sampling errors for visible minorities, low-income groups and First Nations.

By province and territory, the highest response rate was in the Northwest Territories while the lowest was in Prince Edward Island, but all the provinces measured in a tight range around the national average.

Statscan will start releasing the survey results next May. The survey is a collection of social and economic information that is meant to help communities plan for child care, schooling, family services, housing, infrastructure and skills training.