For much of the previous three days, thousands of brokers have had no way to submit applications to lenders. That’s because “Expert,” the industry’s biggest mortgage origination platform, was down.
The outage cost brokers deals in some cases, and prevented many from meeting client conditions, timeframes or appointment commitments. The aggregate industry-wide cost could easily be into seven figures, assuming even one out of 20 brokers lost a deal or a future client relationship because of this. There are also losses to lenders who now face application backlogs and potential lost deals.
Expert’s parent, technology provider Davis + Henderson (D+H), says the outage was caused by two separate issues. The first is suspected to be a database server problem (they’re still investigating to confirm this). The second was a disruption of service by its Internet service provider. D+H says the interruption was unrelated to it switching data centres last weekend.
As of this morning, Expert was back online with some brokers reporting a noticeable improvement in screen loading speed, compared to before the outage.
“Our biggest priority at the moment is stabilizing the environment and improving performance,” D+H VP of Lending Solutions Chris Pornaras told us today.
“We want to ensure that service levels meet what is expected of us before issuing a formal statement to our lending partners.” At that point—likely by the early part of next week—D+H will send out an official report on the incident.
All of this begs one obvious question: Why didn’t D+H have better redundancy? Expert is mission-critical to our industry. The brokerage business almost grinds to a halt without it.
Pornaras says, “We do have fully redundant systems…but because of intermittent service we wouldn’t go to full back-up (on Monday and Tuesday).”
When there is intermittent service, “We don’t call it an outage,” adds Jerry Lo, D+H’s Senior Director, Strategic Accounts. “Our redundancy plan takes effect (only) if everything is down.”
“…You almost always prefer to fix than to go to redundancy,” he notes. D+H therefore kicked in its redundancy plan only when it appeared that it would take longer to correct the problems than to cut over to its full backup system.
D+H would have made that call sooner, Pornaras says, but “Our partners assured us they would rectify (the problems) sooner than they actually did” and the diagnosis and fixes took longer than expected.
Many brokers will nonetheless wonder why downed users cannot automatically be switched over to separate backup servers in a separate data centre with a separate ISP. In 2012, where things like cloud servers and mirroring promise continuous application availability, you’d think there’d be a more instant backup system available.
This week’s problems are not the first for D+H. It has had “a couple of major issues over the last six months,” says Pornaras. March 28 was the last one.
He admits, “Outages are not what is expected of us,” and adds that D+H has made a “significant technology investment over the last several months” to “avoid these types of issues.”
This week’s downtime has led some in the industry to re-evaluate alternative solutions. MorWeb is the obvious one. Kate Raynham, VP Client Services, Marlborough Stirling Canada, tells CMT, “We’ve had a noticeable increase in calls this week from brokers interested in setting up MorWeb accounts.”
(Just before this story went to press, MorWeb emailed the industry, claiming “8 years of 100% up-time.”)
In light of this, we asked Pornaras for his take on whether Expert deserves it dominant market position.
“We believe that competition is actually healthy,” he said. He noted very accurately, however, that alternative “technology has not been able to sustain user demands.”
This is true, and while some brokers rave about MorWeb’s user interface and back-end, its Achilles heel is its inability to route directly to lenders like TD, National Bank, B2B Bank, Home Trust and Bridgewater Bank.
(On the upside, MorWeb has various advantages like allowing all brokers to enter applications offline, including free integrated document management and storage, and imposing no contract minimum on the number of deals that must be submitted on its platform.)
Pornaras admits that there is a perception by some brokers that “D+H just doesn’t care.”
“That is simply not the case.” he says. “Folks have been working 24 hours a day” to fix the problems this week. “Clients don’t see that.”
“We don’t take for granted our customers…We enjoy our (leading) position at this point because we’re a best of breed solution.”
Sidebar: D+H is trying to get better at messaging the community when problems occur. This week, for example, it created a new Twitter handle @DHMtgMktplace to provide status updates to brokers in cases like these. Let’s hope brokers rarely need to use it.