When you’re one of the world’s biggest dating sites, and you’re growing so much that you need to tell your existing back-end infrastructure “it’s not me, it’s you”, what do you do?
You shack up with something bigger, better and able to grow at scale, with eHarmony in the process of doing just that.
It’s all thanks to massive growth in eHarmony’s four big markets - Australia, the US, Canada and the UK.
We interviewed Thod Nguyen, CTO of eHarmony, and his extensive comments are several paragraphs below.
eHarmony had a big challenge - moving to next-gen IT infrastructure, stemming from a need to handle large amounts of big data and rapid growth.
At the same time, the company needed to ‘maintain a service-level agreement of 99.9% to its members with fast response times across all platforms.’
That’s because, on a daily basis, eHarmony’s cutting edge technology creates ‘over three billion potential matches and delivers over 10 million fresh, authentic, and compatible matches worldwide’.
In Australia alone, the company, which uses a compatibility matching system based on 35 years of scientific research that is presumably a zillion times more accurate than Dexter (the compatibility matching robot from Australia's 'Perfect Match' dating show in the 1980s) could ever have been, has more than 2.5 million registrations and has been responsible for over 11,000 marriages since its launch in 2007.
That’s a lot of love.
The company’s tech geeks and gurus quickly foresaw the need to ‘move to a scalable and robust infrastructure that could deliver timely, well-suited and fresh matches, rapidly scale during periods of high volume traffic, and simultaneously, deliver maximum operational efficiency in terms of ROI.’
So, what did eHarmony do? What was its own perfect technological match?
That solution is still in train, with eHarmony moving to an ‘open-standard, flexible private cloud solution’, with this migration set to be complete by the end of 2015.
This migration will ‘cut costs, streamline data centres, reduce complexity, increase efficiency and improve server utilisation’ - all the things any company running on technology wants to achieve today.
It will also enable new development environments managed by a small tech operations team of around 30 people.
eHarmony has built its own ‘infrastructre-as-a-service’, with company operations moving towards a ‘unified computing platform that includes cutting-edge server blade technology to minimise downtime, enable easy provisioning of servers, and integrate storage solutions.’
Most important, at the heart of it all is eHarmony’s determination to stay relevant and on top of cutting edge technologies, innovations and efficiencies.
So, what precisely did eHarmony do?
iTWire was invited to chat with and interview eHarmony’s CTO, Thod Nguyen in the US, which we did by phone conference. I was typing very quickly as Nguyen spoke, so I may have paraphrased slightly here and there, but this should nevertheless be an accurate rendition of what we spoke about - so enjoy!
Nguyen explained that, in the past, eHarmony’s IT system had grown to eventually become dated and inefficient, although after years of faithful service.
Then, back in 2013, Nguyen and his team decided it was time to upgrade to a ‘true enterprise-grade infrastructure’.
A big reason was the need for the ‘ability to process and manage big data but also the complexity of the data model in order to support the growth of the business, both from traffic, user interaction, unique visitor per month and the number of matches eHarmoney processes’.
Indeed, the company typically processed about 2 million matches per day globally 18 months ago - today it is well in excess of 10 million matches per day, from a list of over 3 billion-plus potential matches.
So with that said, Nguyen explained that eHarmony is “very focused heavily on the customer perspective. We want 999 in terms of uptime. 99.9 uptime to customers globally, including Australia, of course”.
Importantly, Nguyen wanted to ensure that eHarmony continued being very responsive under an ever increasing load - “across devices, the mobile web, native apps, devices, iPhone, Android and Windows Phone”.
Very critical, said Nguyen, was ‘dramatically reducing our TCO - at the same time - leverage the most cutting edge technology, IT innovations and a heavy focus on efficiency’.
“We want to be very nimble and lean”, said Nguyen, “we have lot of requirements we needed to address as we move to a truly enterprise with high scalability and high availability”.
Nguyen explained that the process started in 2013, using a different type of critical technology from the application perspective, and moving towards a service oriented architecture, with the concept of cluster services where every service can be scaled independently - all without impacting the website”.
“We’re aggressively moving to an SLA model”, said Nugyen, with “the biggest impact being the ability to build the foundation part.”
“So, from the foundation persepctive, we are aggressively replacing all the architecture from network architecture and moving towards an on-premise private, open cloud solution - based on Open Stack.”
“We have reduced our servers by 40% while supporting the same traffic” and “we use next-gen server architecture powered by Cisco UCS - ‘united computing system’, which allows the solution to manage competing resources but also storage under one centralised fabric component.”
“So, the underlying server architecture used by Cisco UCS - on top of that the KVM hypervisor - enabled us to increase the cloud computer very quickly as we scale - we can easily add 10-15 nodes very effectively.”
“The goal”, said Nguyen, “is to able to do that very similarly - Amazon AWS for example on a much smaller scale - because of the cost - but also the ability to manage the solution in a much more secure way - which is related to our efficiency and operational cost reduction.”
“Those are the two biggest components”, said Nguyen. “We also aggresively moving towards supporting a next-gen advanced analytics solution which allows us to collect a lot more data - we have so much data right now and we the same centralised data - different data from more data sources”. It gives eHarmony “insight into customers - improved match quality - and keeps the customer very engaged”.
“We are truly a data driven company”, said Nguyen. “machine learning - big data - we’ve been using it for a long time!”.
So, it was “time to take it to the next level, [as] our data model is a lot more complex.”
“We need the right application and infrastructure to do a lot of complicated data analysis and statistical analysis.”
“Those are the two biggest things we have deployed. It is our next-gen refresh. We expect to complete the entire process by January 2015.”
“We have started the whole proof of concept, planning, deployment as we go for the last 2 years”, with another phase to go over the next couple of months “before we are complete”.
“We manage about 1000 servers now on premise and a total of about 3000 devices that we manage in the primary data centre- those are the high-level key benefits and requirements and what we expect the IT infrastructure to be - To really deploy a solution to run the most complex big data over the next decade”.
Nguyen explained that “it’s not just about addressing today but by forecasting what the biz really needs to accelerate and expanding the same underlying infrastructure to support other verticald as well.”
“You may have heard of our new elevated career job compatibility - which is going live in 2015, which will leverage what we have built to other sectors.”
So, said Nguyen, the “key things - key objectives - translating, reducing the operating cost or expenses, shifting operating expenses into capital expenses. Reducing operational costs, saving costs through this solution and move those expenses towards the innovation part - capital expenses, because for us to be very competitive in the marketplace we need to focus strictly on innovating and less on day to day operational costs. That’s what we want to be - one step ahead of our competitors especially as technology evolves very quickly.”
“We spent about 6 months in 2013 to revisit what we have, all the requirements for the business and forecasting what we think the business will be 3-5 years from now. A combination of an enterprise solution but also an open source solution as well, depending on the requirements - solving complex problems with an optimal solution and being able to roll out the solution very quickly without zero impact on customers.”
“Thats our key no 1”, said Nguyen, “providing a top quality product to our customer but also the highest availability with top performance - especially on mobile devices. That’s very challenging. Very complex from 3000+ devices from one to the next without any glitches or impact - our customers will always be our no.1 focus!”.
“That’s adding anther dimension of complexity - massive data - a data model that’s extremely complex. We take 29 dimensions of compatibility into our 150 question questionnaire that we ask our customer to fill out - you can just imagine - take that exponentially - in terms of complexity, traffic and the same time ensuring that our customer can access to the website has 99.9% uptime”.
I asked Nguyen more about the process, about whether the old system and the new are running in parallel and more.
Nguyen responded: “We have phases - we’re replacing our server infrastructure incrementally - any system we roll out - we do things in parallel. It’s very well planned out in terms of deployment - shadowing system, replicated - all before we turn the switch on, we incrementally roll out one component at a time. As an example, we recently moved to flash storage”.
“Here, we have a next-gen data warehouse solution powered by IBM’s Pure Data Neteeza system data warehouse. It processes data 4-5 times faster. We are moving our network architecture, firewall architecture, database, system application architecture, storage and moving a lot of stuff to focus on cloud compute on premise, with virtualisation tostrealime the process and greatly reduce our cost of operation”.
“We use it for a lot of data mining and analytics, so we actually move every single component one at a time, depending on the impact and also the risk. Everything we do is running in parallel so we can test the production traffic and start moving one piece at a time until we are completely done. This has been the approach for the last 1.5 years.”
“We needed extremely careful planning, it has to be flawless. There’s no way to go back! It’s very different to rolling out an application - you can go back. But in the case of complexity of infrastructure - we have a complex distributed system - from application database, storage, network - everything is fully redundant, distributed”.
“The key thing is allowing us to be very nimble in terms of rolling out new solutions - at the same time to be able to adopt new tech very quickly - that’s the biggest tech advantage.”
“We can easily expand it to other verticals without revamping the whole system from scratch - that is creating a big competitive advantage against competitors. We focus on a lot of automation, the ability to adopt new tech very quickly - and we build something that will be around for several years to come.”
“We will always improve the system as we start seeing new solutions - we have built a plug and play concept. If say four years from now, a new vendor builds a better solution which one of our current partners can not yet match, we’re not completely coupled - we can negotiate - we can expand to other upcoming solution providers”.
“We have architected in such a way that not really tied to one vendor - we can evolve or we can replace. If our partner are not innovative and someone else is, our focus will be to partner with the most innovative companies to be able to help us to move quickly to address business needs, controlling our efficiency and reducing opex vs capex.”
Nguyen emphasised again that “the biggest piece is innovation and efficiency to accelerate not only our business growth, but to provide the best possible product to our customer”.
It sounds like a harmonious match made in heaven!