Building a video advertising infrastructure that dynamically scales to support hundreds of publishers, advertisers, trading desks, enterprise partners, and demand side platforms has been core to our success at LKQD. Today, our platform processes 5 million requests per second and logs in excess of 800 billion data points every day and is supported by 45,000 CPU cores. We circled up with our CTO, Christophe Clapp, and VP of Engineering, Andrew van Dyk, to get their insights and expertise.
In an industry that’s always changing, how has LKQD grown its’ infrastructure to scale with customer needs?
Christophe Clapp: My main goals have always been to ensure that we continually innovate and to develop high performance, reliable technology. This focus has been key to ensuring that LKQD continually leads the video advertising industry.
Andy van Dyk: LKQD has grown tremendously since it was founded. My job is to make sure that our infrastructure has seamlessly scaled as we’ve grown, and continued to deliver optimal performance. We started in the cloud where it’s easy to scale quickly as you grow your customers. Now we have around 1,000 dedicated servers, running on CoreOS, Kubernetes, Nginx, and Ceph backed object and block device storage.
What are some of the key challenges to building a real-time, scalable infrastructure?
CC: From the start, LKQD built our technology to be scalable and adaptable. It’s been one of the keys to our success and propelled us to the number one comScore* ranking among video ad platforms. We use a hybrid cloud infrastructure that enables us to maximize the reliability, scalability, and efficiency of our infrastructure. Managing this way is more efficient than building our own data-centers, and it gives us greater agility along with unlimited scalability.
AVD: When I started we were doing a couple hundred thousand requests per day. Now, we process millions of incoming and outgoing requests per second. We generate hundreds of billions of rows of event logs each day, which are aggregated by many different dimensions in near real-time. Being able to scale quickly has been key, as each component of the software and hardware stack has an upper threshold for capacity. One key challenge was cost management, as it can quickly spiral out of control in the cloud. To be financially viable, initially we were using spot instances. Now we have fixed costs and more stability by having our own managed data center.
How did the infrastructure evolve over time?
AVD: Initially, we managed our platform through cloud-based service providers, and decided to migrate to a hybrid cloud infrastructure in 2016. We were heavy users of spot instances, being able to run servers at a discounted hourly rate of up to 80%, however there were stability issues with this as large amounts of spot instances could be taken away at any moment. Running in a dedicated managed data center gives us peace of mind that we don’t need to rely on spot instances, and we’re able to burst into the cloud if we need extra capacity.
CC: We started with a credit card and a handful of spot instances and grew to over 5,000 instances within 18 months. This approach allowed us to scale up quickly without commitments. However, as our infrastructure and bandwidth usage continued to grow, we realized that the savings we could achieve with managed colocation were significant. We spent time doing research and speaking to various cloud and colocation providers. Our conclusion was that Rackspace let us deploy Kubernetes on bare metal without having to worry about inventory management, networking, and hardware replacement — all at a cost that came very close to managing everything ourselves.
Why choose managed provider vs. building your own data center?
AVD: There’s a lot involved in building and running a data center, from the beginning stages of the overall design, redundancy, racking servers, cabling, replacing bad hardware, monitoring power and usage. Handing this off frees up an immense amount of time for us to focus on the business and product stability.
CC: We knew we needed a company that had a lot of experience and expertise in building data centers. We worked with Rackspace partly because they are the world’s leading provider of IT as a service, and help us tap the power of cloud computing without the complexity and cost of managing it on our own. This frees us from the worry of managing a large scale data-center environment, and enables us to focus on the business and product. We are also a lot quicker. In our new environment we can scale any application within seconds to the required amount of instances to handle demand — and this gives us a competitive edge.
In summary, ensuring the LKQD platform remains flexible, scalable, and reliable has been a key to our company’s success. Continually expanding and evolving our infrastructure within an industry that’s constantly changing has helped us establish and maintain a leadership position and ensures that we meet the needs of customers.
Read more about this in our case study: LKQD Rackspace case study
*comScore Video Metrix®, Video Ad Ecosystem Rankings, August 2017