By Will Dunning, Special to CMT

---

The bottom line: Statistics Canada’s estimate that Canada lost 31,200 jobs in July is probably wrong. The agency could improve the reliability of its employment numbers by increasing the survey sample sizes, particularly in Toronto and Montreal.

*****

On August 4, we were told that “Canada lost 31,200 jobs in July”. It would be much more accurate to say “Statistics Canada has estimated that Canada lost 31,200 jobs in July”.

This country’s employment data comes from a sample survey, and like others (such as political opinion polls) there is a “margin of error”. The numbers produced by this survey are estimates, they are not reality.

Consider the following:

• Statistical theory tells us that 5% of the time, the estimate of monthly job growth will be off by “two standard errors” (or more). The standard error for July was 29,000.
• As a result, while July’s estimate was -31,200, the “true” figure is probably somewhere in the range of minus 91,000 to plus 28,600. 
• Alternatively, a more simplified way to view this is that “95% of the time the estimates are accurate”. On this basis, we (the public at large, the media, and indeed the vast majority of economists) accept the monthly numbers as objective truth. It is exceptionally rare for economists to publicly state that Canada’s monthly job creation numbers are unbelievable.
• Statistical theory, however, also tells us that 32% of the time, the estimate will be off by one standard error, or 29,900. In other words, given July’s estimate of -31,200, there is a 68% chance that reality is somewhere in a range from -1,300 to -61,100.  There is also a 32% chance that the reality is either a loss of 61,100 jobs (or worse) or a loss of 1,300 jobs (or better, and possibly an actual gain).

To put this into further perspective, in a healthy economy (in which employment might grow at the same rate as the population), we should currently expect to see job creation at approximately 23,500 per month. If there is a 32% chance that the reported estimate will be off by plus or minus 29,900, there is a big risk that StatsCan’s monthly reports will give us highly misleading pictures. Jobs might, in reality, be created at the healthy rate of 23,500 per month, but one third of the time the estimates will show either job losses or very strong growth.

On this basis, whenever we see surprising numbers on job creation, we should always ask ourselves how much we should trust that data. The July number of -31,200 was indeed surprising, and there are strong clues that this estimate should not be trusted.   

 

Too Many Surprises in Canada’s Three Biggest Cities

In the Labour Force Survey, Statistics Canada publishes estimates for Canada, the provinces, and the largest urban centres (Census Metropolitan Areas, or “CMAs”), as well as for various “economic regions”.

The three largest CMAs (Toronto, Montreal, and Vancouver) account for 36.3% of the population that is covered by the survey.  Therefore, any significant changes in the numbers for these three urban centres have a big influence on the estimates for Canada.

Correspondingly, when there are significant statistical anomalies in their data, they often result in statistical anomalies for all of Canada.

Since the start of 2012, these three CMAs have created jobs at an estimated average rate of 10,800 per month. This is 81% of the estimated rate for all of Canada (13,300 per month). But, as can be seen in Figure 1, the reported data is highly variable from month to month. Many of these estimates are so high or so low that they appear quite unlikely. 

Figure 2 contrasts the growth estimates for the three large CMAs with the entire country. In this chart there are 55 months of data. Of the 55 estimates for Canada, 17 are surprising: eight are surprisingly high (being above 40,000) and nine are surprisingly low (below less than -15,000). 

All told, 31% of the estimates for Canada are surprising, which is similar—interestingly enough—to the 32% we expect based on statistical theory.

The data in Figure 2 shows that out of the 17 surprising employment numbers in Canada, 10 are associated with unusually large changes in one or more of the three CMAs.  

The most recent example of this was last month. For all of Canada, there was an estimated loss of 31,200 jobs. For the Toronto CMA, there was an estimated loss of 24,300 jobs, which would be a drop of 0.7% in just one month. This is an enormous change for such a short period, and it is not believable.

The previous case where one of the CMAs distorted Canadian data occurred in October 2015. In that instance, StatsCan’s estimate was for a rise of 43,100. Vancouver CMA had a reported gain of 22,800, (1.7% in just one month). Again, this was not believable.

The Implications

If Statistics Canada could reduce the erroneous variations in the employment data for Toronto, Montreal, and Vancouver, they could improve the data for all of Canada.

The way to reduce the variation is to increase the sample size. As it stands, the sample sizes for the three CMAs are too small. The table below shows that in combination, they account for 12.5% of the national sample (which is about 56,000). Their combined share of the sample is only one-third of their actual proportion of the population (the 36.3% shown in the table below). Moreover, Toronto and Montreal are more “under-represented” than Vancouver, and correspondingly they more frequently distort the national numbers.

It isn’t necessary for each CMA to have a sample share that matches its share of the population – the largest of them can reasonably be under-represented to some extent. The point here is that their under-representation is too severe and this is unnecessarily diminishing the data quality for all of Canada.

---

Will Dunning is the Chief Economist for Mortgage Professionals Canada, as well as operating his own consulting firm, which specializes in analysis of housing and mortgage markets. For Accredited Mortgage Professionals, his presentations on “Analyzing and Understanding Canadian Housing and Mortgage Markets” qualify for 1 CEU credit in the compulsory category.