emili Criticisms Resurface
The Globe and Mail has taken another look at automated home valuations, casting a dark shadow on their efficacy. It’s the paper’s second take on this topic since October. (Here’s the prior story: CMHC’s emili Under Fire)
Like the last go-round, this story is laced with concern. Some of that concern is justified given that:
a) housing finance relies on automated underwriting systems (like CMHC’s emili and Genworth's Excel platforms), and
b) those systems operate with a high degree of obscurity.
But like most articles on automated valuations, this one is devoid of data that quantifies the risks being dramatized. To judge the benefits and risks of automated valuations, the public needs more context, and that can only be conveyed with data. We must answer questions like:
- What ratio of properties do systems like emili “overvalue” versus human appraisers (who certainly aren’t perfect in their own right)?
- How much are those properties overvalued, compared with in-person appraisals?
- What is the conservative projected default rate for overvalued properties, given a housing selloff?
- What is the potential severity (loss amount) of these defaults?
- Are insurance premiums sufficient to offset those losses?
- What is the net economic value of low-cost, instant and objective automated appraisals (as opposed to not having them)?
- What do these defaults have in common and can underwriting bots be better-"trained" to spot those risks?
The problem is, much of the data required to answer the above are highly sensitive for competitive and PR reasons. Yet, it’s this type of analysis that’s mandatory for an objective risk assessment.
The Globe has indeed raised valid questions about automated valuations (none of which are new questions—these matters have been debated since the 1990s). Now, it is the regulators’ responsibility to use their power, summon automated valuators to provide more data, analyze the actual risk, and then share its findings with the public. We, as the public, have a right to know how systems like emili perform, because it’s our economy and taxes on the line.
But until that happens, let’s be careful about damning a system that’s added tremendous efficiency to the financing process for well over a decade. We must also remember that individual overvaluation isn’t the issue. The Globe piece features anecdotes about specific cases where emili got it wrong. Yet, the real issue from a systemic risk standpoint is the overall portfolio performance of emili.
Over portfolios with hundreds of thousands of properties, there will always be overvaluation and undervaluation, and the overwhelming majority of those cases fall within safe parameters. What the public needs to know is the tail risk of auto-valued applications, versus appraiser-evaluated apps. Unfortunately, we don’t have enough data to gauge that yet.
Inevitably, people will read the Globe’s story and think that CMHC is using some back-of-the-napkin formula to judge property risk. That’s so far from the truth. Emili is not some 100-line computer program written by a college intern. It is multi-million dollar mission critical technology benefiting from the best available data and over two decades of R&D.
CMHC knows the risk of it botching property valuations en masse. It has the public, press and regulators breathing down its neck around the clock. It knows the risks in automated valuations better than virtually anyone in the country because it’s processed millions of mortgage files since 1996. And it tirelessly optimizes its systems to statistically factor in and adjust for those risks. To imply that CMHC cannot account for “recent movements in home prices” is simply laughable.
For now, automated valuations aren’t going anywhere. Their objectivity, speed and cost savings simply add too much value to be discarded. But we’ll certainly hear more debate about them in the coming year.
Rob McLister, CMT
