emili Criticisms Resurface

appraisals-vs-emiliThe Globe and Mail has taken another look at automated home valuations, casting a dark shadow on their efficacy. It’s the paper’s second take on this topic since October. (Here’s the prior story: CMHC’s emili Under Fire)

Like the last go-round, this story is laced with concern. Some of that concern is justified given that:

a) housing finance relies on automated underwriting systems (like CMHC’s emili and Genworth’s Excel platforms), and
b) those systems operate with a high degree of obscurity.

But like most articles on automated valuations, this one is devoid of data that quantifies the risks being dramatized. To judge the benefits and risks of automated valuations, the public needs more context, and that can only be conveyed with data. We must answer questions like:

  • What ratio of properties do systems like emili “overvalue” versus human appraisers (who certainly aren’t perfect in their own right)?
  • How much are those properties overvalued, compared with in-person appraisals?
  • What is the conservative projected default rate for overvalued properties, given a housing selloff?
  • What is the potential severity (loss amount) of these defaults?
  • Are insurance premiums sufficient to offset those losses?
  • What is the net economic value of low-cost, instant and objective automated appraisals (as opposed to not having them)?
  • What do these defaults have in common and can underwriting bots be better-“trained” to spot those risks?

The problem is, much of the data required to answer the above are highly sensitive for competitive and PR reasons. Yet, it’s this type of analysis that’s mandatory for an objective risk assessment.

CMHC-emiliThe Globe has indeed raised valid questions about automated valuations (none of which are new questions—these matters have been debated since the 1990s). Now, it is the regulators’ responsibility to use their power, summon automated valuators to provide more data, analyze the actual risk, and then share its findings with the public. We, as the public, have a right to know how systems like emili perform, because it’s our economy and taxes on the line.

But until that happens, let’s be careful about damning a system that’s added tremendous efficiency to the financing process for well over a decade. We must also remember that individual overvaluation isn’t the issue. The Globe piece features anecdotes about specific cases where emili got it wrong. Yet, the real issue from a systemic risk standpoint is the overall portfolio performance of emili.

Tail RiskOver portfolios with hundreds of thousands of properties, there will always be overvaluation and undervaluation, and the overwhelming majority of those cases fall within safe parameters. What the public needs to know is the tail risk of auto-valued applications, versus appraiser-evaluated apps. Unfortunately, we don’t have enough data to gauge that yet.

Inevitably, people will read the Globe’s story and think that CMHC is using some back-of-the-napkin formula to judge property risk. That’s so far from the truth. Emili is not some 100-line computer program written by a college intern. It is multi-million dollar mission critical technology benefiting from the best available data and over two decades of R&D.

CMHC knows the risk of it botching property valuations en masse. It has the public, press and regulators breathing down its neck around the clock. It knows the risks in automated valuations better than virtually anyone in the country because it’s processed millions of mortgage files since 1996. And it tirelessly optimizes its systems to statistically factor in and adjust for those risks. To imply that CMHC cannot account for “recent movements in home prices” is simply laughable.

For now, automated valuations aren’t going anywhere. Their objectivity, speed and cost savings simply add too much value to be discarded. But we’ll certainly hear more debate about them in the coming year.

Rob McLister, CMT

  1. In a hot housing market like Toronto there is a big difference taking 3 days for an apprasial and 3 seconds. Appraisers hate emili because it eats their lunch, but it does assist home buyers.

  2. I disagree. It’s this kind of “oh well, it’s not off by that much,” thinking that could send us into a tailspin. How does it help homebuyers when their home isn’t appraised correctly? It may help them when they’re allowed to borrow too much equity; but as we know, this will hurt them in the long run. Emili is not a good system, and there’s got to be a better way. I don’t typically hold too much stock in what The Globe says, but they were bang on in this case.

  3. “It is multi-million dollar mission critical technology benefiting from the best available data and over two decades of R&D.”
    — uhh…. it may be multi-million dollar system but big $$$ doesn’t mean it is programmed well (gun registry anyone???) or that it is efficient. I have first hand experience with the internal logic of the system and I can say that it does not use “best available data” nor does it look anything like a system benefiting from “2 decades of R&D”.

  4. Appraisers offer subjective valuation assessments. Emili uses extensive reams of data to arrive at a valuation. Neither Emili or appraisers are perfect in their valuations and never will be since the true value of a property is what a buyer is willing to pay in a open market.
    Emili has been around a long time and CMHC stands behind it in spite of what the almighty Globe & Mail might have to say about it. That is good enough for me.

  5. CMHC has been a trusted source of Canadian market data for many decades. Such reports continue to be reliable, valuable and reported on by many distinguished organizations today.
    To suggest that by your assessment and personal experience (Where’s your credentials/facts) they don’t know what they are doing is pure slander.

  6. Tell us more about your “first hand experience with the internal logic.”
    If emili’s data is not the best available, what data is better?

  7. I too would enjoy stories of your supposed “first hand experience with the internal logic.”
    It’s a safe bet you’re not on CMHC’s emili development team, which makes your experience hearsay at best.

  8. That is a refreshing and enlightening article on the subject. This is what I consider to be a sober, objective, logical approach to a potentially explosive issue.
    The writer obviously recognizes the massive importance of this very sensitive issue, but prudently avoids all the hype that usually characterizes sensational reporting. The writer correctly sets out the research work that needs to be done before we declare the issue to be a problem, something which previous articles have failed to do.
    This Appraisal issue s not a problem yet, but some of us want to score points by being the first to declare it a problem.
    Let’s get the facts first and then we will decide, based on the facts, that it is a problem that needs to be addressed.

  9. I think we all know that the real problem is the size and scope of CMHC’s activity relative to the size of the market. Whether it’s appraisals or pricing or rationing of the insurance the magnitude of the top-down central planning will be its downfall.

  10. The similarity between the CMHC’s overly-optimistic model (emili’s inputs come from sellers according to the Globe) and the ratings agencies’ overly-optimistic models leading up to the financial crisis in 2008, are frightening in my opinion. When risk is underestimated and a country’s financial system “doubles down” on what looks like a safe bet, the results can only be catastrophic.

  11. How do you actually know that emili’s model is “overly-optimistic?” You probably know little apart from what you read in the Globe and Mail. Do you really think that a system used to evaluate more than half a trillion dollars in real estate has only seller information for “inputs?”
    Come on people. Ask logical questions and don’t form judgements with only half the story.

  12. Quantifying the tail risk would be an interesting exercise. The article seems to indicate that there is the potential to “game” the Emili system in cases of refinancing -Especially for over extended and vulnerable borrowers.
    It hard to draw any conclusions whether abuse would be more prevalent using Emili vs. manual appraisals.
    In the past, my work involved dealing with insurance companies seeking replacement items for their clients. It was rather surprising how many adjusters demanded personal “off the book” compensation in exchange for sending over their business.
    I’ve also had similar experiences with buyers purchasing promotional items on behalf of charities.
    It seems too often, human intent is not altruistic in nature and weaknesses will be exploited.

Your email address will not be published. Required fields are marked *