Replay-Building and Managing Multicloud AI Infrastructure

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

All right, you're listening to another episode. Sean, you absolutely asked me if this episode of the CTO Advisor is video or audio only, and it's audio only. Sean, introduce yourself. Hey everyone, it's great to be back on the podcast. I had a lot of fun the last time we chatted. My name is Sean Mullaney, and I am the CTO at Algolia, who is one of the world's largest search as a service company. You know, you teased me. I didn't know if Algolia was a country, small country, or if it was a software company.

Well, we are the world's second biggest search engine behind Google, so we kind of serve enough traffic to be a small country. So we teased this episode during the last sponsored podcast. So this isn't sponsored for those of you who listened to the podcast that we talked about, kind of an introductory to AI-based search. Great podcast. It goes over the basics of AI, talking about vectors, all the way to how you defeat hallucination and search for products, et cetera. Great podcast, go listen to that.

But I think we wanted to hit two topics in this podcast in the follow editorial podcast, which was going from a software culture to AI culture. And then secondly, going to a multi-cloud operating model, which intrigued me during the last conversation. So Sean, first off, most of us listening to this podcast have software houses. We understand the CICD pipeline. We've either there, we're developing that, we're getting to a point where we're treating our infrastructure as code. Software development is starting to become, or has become this table stakes for most IT shops.

Now we're being disrupted and we can expect some disruptions as part of the industry with AI. How should we start thinking about AI versus software development, or AI development versus software development? Yeah, it's a fantastic question. And I know a lot of people are going through this at the moment. So there's a big difference between running a software engineering cultured company and running a company and a team that is building AI software. And I'll talk to you about a few of the major things that are the difference.

Firstly, when you think about software, most of the time you're able to scope out both the problem, the technical requirements, what are the acceptance criteria for the user in advance? And you can kind of use a kind of waterfall model where you design the solution, you put the Jira tickets together, and you can estimate how long it's going to take for the project to finish. And you can do this across a whole bunch of projects and you have a certain amount of certainty that when you put in a certain amount of investment, you'll get a certain return for that based on the software's value.

So it feels quite deterministic once you get a good system in place for your team. All of that changes when you're building AI software. Firstly, you're not really able to in advance estimate how long it's going to take to train an algorithm and to continually iterate in production on it. So first thing is, is you can do a prototype, but every single time you want to test it, you've got to put it into production because the only real results you can get back is on real production data.

And you've got to go through a very fast experimental scientific process. So you think a bit about how scientists do R&D is they have a hypothesis, they then go and test the hypothesis. In this case, you might test some new features, a new algorithmic approach, and then you go and get your data. And then you analyze the data and you've either proven your hypothesis and improved or your hypothesis was not proved. And your progress is a function of the speed and the velocity of how you experiment.

And so it's not as deterministic as something like designing a piece of software where you can do that in advance. So definitely some pretty big differences when you're thinking about approaching a kind of AI problem versus a traditional software problem. So let me think about this from kind of my infrastructure lens. We deal with this in load testing. Typical problem I faced earlier in my career, we did what seems like table stakes today, but we're streaming video to 12,000 clients. We're spread across the US.

I could load test the system over a LAN and we could put 20,000, 25,000 robotic clients against that system. But what we couldn't simulate is the diversity of the clients across 200 different sites, different wind speeds, et cetera. Infrastructure is like that. You can't test infrastructure for load. AI, is it a similar challenge that you can't test for the variety of data? You have the data that you train your model on, but when you put it in production, you can't simulate the scale of other people testing against that dataset.

Yeah. I mean, when you develop software, you have like two hypotheses, right? One, that it's gonna work like you designed it and that you have unit tests, you have integration tests, you have user acceptance testing, and you can kind of test that. And then secondly, does it add the value that you'd hoped it had for the customer? In AI, you have kind of like different problems here. So you can do offline testing. So you can take a snapshot of user interactions and data.

Let's say we're testing a new ranking or sorting algorithm for an e-commerce store. Sure, you can do some offline testing, but what we found is that this offline testing is only a kind of indicator that gives you a certain level of confidence that the algorithm should perform better. What you find out is when you put it into production, you get results that are often differ quite significantly from the offline testing. And the reason is, is because user's behavior changes, particularly when you present a new algorithm or a new set of outputs to them.

The old data that you've captured before is working on the old algorithm. You produce a new algorithm. Your offline testing was the old data. And so these algorithms change the behavior of users. So it's actually, it's helpful to give you a kind of directional indication of whether the changes you're making are gonna be positive, but you've got it. The only way you get real data is by testing in production. Wow, so you're inadvertently answering a question I asked. It was either last week or earlier this week, week one LinkedIn, which was the effectiveness of AI for testing.

So looking at this from a different angle, one of the problems we have with application testing in general is user inputs. So we can't simulate a marketplace, not today. I can run back the last transactions from the past three months. But as you pointed out, that doesn't really account for when I change the interface. If I change the interface, maybe the way that the transactions were made for the past three months won't play exactly the same way in the new software design or the new system design, the new interface, users are different.

So the question that I posed was, well, can we use AI to solve this problem? And I think that secondary problem is yes, you can kind of use AI to create chaos, but you're still not going to get a representation. It's not real people. So you're not going to get a representation of how real people are going to engage with the system. Yeah, so the new way in which you experiment and develop software is by having consistent A-B testing all the time.

When you've got a certain amount of traffic, the traffic itself is just a huge gift to the software development team. And you actually have to say like, let's say we have five or 10 ideas that we're developing right now. We've got to get an A-B test set up for each of these. The more traffic we give each experiment, the faster they're going to converge on a result. And we're going to understand whether their experiment's going to be successful or not. So often what I've seen is, is that you've got to allocate traffic to teams in order to speed up or slow down their experiments.

And often your bottlenecks, the number of experiments you can run simultaneously by the amount of traffic that you can allocate. And so, you know, when you're in this AI development world, it's all about the velocity and number of experiments that you can run on the traffic and the production data that you have. So let's talk about that rework, because in the A-B, in the software world, when I'm doing A-B testing, I can rapidly adapt to and change and update my code based on that A-B testing.

This is the, you know, new world that we live out of, that we can, you know, deploy multiple times throughout the day and get this right. Yeah. In the AI world, I would imagine that retraining or adjusting a model based on inputs, is that, you know, is that one of the areas where we're running into friction and determining how long an AI project will last or take? Yeah, and that's another, so you not only have the development phase of an AI algorithm where you're obviously, you know, making changes to the algorithm, making changes to the data to figure out what works well.

But once you actually launch something into production and you're like, okay, this product's ready to go live. We've A-B tested it. We've got good results. You then have to, as a kind of production serving dev ops, keep the data fresh every day and retrain the model. And so typically these models are trained in batch overnight, redeployed once a day, but then you have to like, make sure that they're performing in ways that you're expecting, because the data that goes to train them changes every single night and their behavior might change every day.

So the observability around AI algorithms and how they're performing is also very important. And I guess that batch process is one of the long tails, right? I make a change. I have to run the new algo against my existing dataset overnight, then make that change. So this ability to rapidly deploy throughout the day gets gated by how quickly I can retrain my model. Yeah, exactly. And typically the retraining of the models moves the window on the data. Let's say you want to train on the last seven or 14 days worth of data.

So every day you've got a new dataset to train on. It requires a lot of GPUs typically. And so what folks do is they go overnight when traffic is very low on the whole time zone that they're operating in, and they might use spot instances and things like this. So retraining the models at two in the morning, you tend to find that the cost of the info has gone down. There's not as much demand for it. So nightly training of these models, when we're all asleep, the models are all training.

I like that. Yeah, as a runner, I can appreciate that analogy. So this kind of leads us into the second conversation, which is around multi-cloud and cloud infrastructure. I'm an infrastructure folk, right? And I can't help but to talk about infrastructure when we talk about massive amount of data, massive GPUs, CPUs. A lot of the pushback I've been getting from end customers around AI has been the investment in infrastructure. So as a AI SaaS provider, it makes economic sense for you folks to make that investment.

But when I talk to enterprise customers, they're like, I'm just recovering from my Hadoop investment. Hadoop, this is, we're not talking about AI, but my Hadoop investment. So they look towards the cloud, obviously, for this type of activity and development. However, what caught my ear the last conversation was that you folks are in all the clouds. Yeah. Or all the major clouds, at least, and some of the minor clouds. And the first thing that jumped out to my thoughts was skills.

Yes. How are you operating across multiple clouds? Yeah, and I'll give you a little bit of a background. So Algolia started just over 10 years ago. And when we started, we were providing very high performance search infrastructure globally to, we have 17,000 customers all over the world. And so what we do is when we started, there wasn't a sophisticated and highly liquid cloud environment. So you went to AWS or even Microsoft or Google back then, they had a very small number of VMs with very limited configurations.

And it turns out that to run a high performance search engine, you need very specific type of VM and memory configurations, which just weren't available in the cloud at the time. So we actually built our own systems inside data centers all over the world, right? We've worked with data center providers. I think we have like several dozen data center providers, thousands and thousands of machines. And we built out this infrastructure ourselves and operated ourselves through data center leasing and partners. Until we got to the point where obviously we're transforming the product into an AI first product.

So all of the search product now is run off of large language models, our re-ranking, our query categorization, all these things are AI models. And AI models are really tough to run in data centers because there's not an elastic environment to be able to leverage. And so as we started to move to the cloud, what we realized is there was another big trend happening. And this trend is that for every one of our customers, we're selling them infrastructure. They have a preferred cloud.

Some of them are Azure customers, some of them are AWS, some of them are GCP, but each of them wants a couple of things. They want to have low latency between the infrastructure services they buy. So they want us to be deployed really close to where they are. Secondly, they want to minimize network egress and ingress charges between the application and their data center and any vendors like Algolia that they're interacting with. Because you've got to pay a lot of money to have the network traffic go in and out of your cloud.

And then thirdly, and this is I think probably one of the bigger things, is that our customers tend to have large multi-year enterprise purchase agreements with the cloud companies. So with AWS, they might be, I think it's called EDP, where they've committed a certain amount of spend over a certain number of years. They want to be able to use those credits to buy from vendors like Algolia. And they can do that through the AWS marketplace, the Azure marketplace. And a lot of these folks made very large commitments during COVID and have seen recently that they're underspending on those commitments.

So we know that we need to be in every single cloud because A, that's where our customers want it from performance, but also from a cost perspective, they want to be able to bundle and buy under these big enterprise deals. So that's kind of a little bit of the history and the need why customers really want to buy in multiple clouds. Yeah, that's pretty consistent with what I've heard. One of the things that I've also heard, just a tangent a little bit, is because customers have this committed spend with the clouds, they're trying to figure out a tough problem.

Like if I'm training models overnight using thousands of GPUs in a specific region, then building a data center doesn't make sense. However, if I have a fairly static workload, let's say SAP, I'm not developing against this. There's no CICD pipeline for SAP. It is just a static application. Maybe once or twice a year, I might have to upgrade the number of CPUs to it. But this is something that belongs in a data center. Use pattern is just different. What I'm seeing is customers who are committed to a minimum cloud spend, they're trying to figure out, they're on this journey, this multi-cloud or hybrid cloud journey.

And they realize, you know what? I probably shouldn't move this application. I have no intentions of upgrading or changing my development motions to the public cloud. Let's keep it on-prem. So they're having a tough time balancing their commitment across the clouds. So as a CTO who obviously needs to optimize cost across three different clouds, how are you managing the problem of workload placement? Yeah. There's kind of customer demand, and then there's just your steady state of where you should run just steady state workloads.

Yeah. And it's not just workload. The other big thing to consider is where do you consolidate your data so that you can train learning models because you want a single data set that you can train in a single cloud. So therefore you're having to move data around between clouds as well and consolidate it into a single warehouse. So it gets even trickier than that. But one way that we think about this is you have customers who are, let's say cloud native, and they're passionate about buying your product in their cloud of choice.

And then you have ones who are cloud agnostic, and they don't really care. You know, it could be in any cloud. They don't mind. They just want a good price and a good product. Our kind of gut feeling is about 80% of people are like cloud agnostic, and 20% are really cloud kind of native and want to make sure it's local. And so we've come, we actually have like a lot of the benefit of coming out of data centers. We wanted to make sure we could run in all the clouds.

So we're very much working in containers, working with Kubernetes and other technologies that are supported across all the clouds so that our workloads are easily moved between them. We can move customers between clouds. We can, in particular, have a lower DevOps having to run between each of these different clouds. But then we, you know, sometimes we have some issues where it's like, there's a managed service, for example, we'd really like to use to simplify our lives. Sometimes it's not great having to manage and run DevOps for everything.

So then we try to figure out how to abstract away the cloud-specific implementation details of each of these, whether that's like, you know, object storage or like database as a service, these kinds of things. But it gives us huge leverage to be cloud mobile, right? We can move workloads between clouds. And it means that when we have negotiations, pricing, regional availability problems, you know, some of the clouds don't have a great availability in one region and another cloud does. I know in Black Friday, for example, you get a lot of customers who are going in and they're buying up all the instances ahead of time.

And you can find yourself like lacking instances in one cloud. So having that kind of mobility gives you a lot of optionality in both negotiating with each of the cloud vendors, moving your workloads to the cheapest or sometimes the most available or liquid pool of instances. And I think it's actually really good for your customers as well, to be able to have that cloud-native experience when they need it. But- Yeah, so from, I love the approach that you're kind of abstracting away from the clouds in general where you can, makes sense.

Yes. But there are areas where you can't abstract away. From a governance perspective, have you formed kind of a platform engineering group to handle that, you know, to maybe add a shim between the clouds and your developers focusing on solving the business challenges and then another team kind of providing a developer experience. So that they're not thinking about which cloud is this gonna run on and more about the problem. Correct, yeah. We've built in our SRE team, we actually have several different layers.

We have an infrastructure as a service team who are dealing with the bare metal and the differences in bare metal between different clouds. We then have like a platform as a services team that are providing the application layer, agnostic kind of interfaces. And then we obviously have the application teams that are serving and running each of the different applications. So we have kind of created a kind of three layer abstraction inside the company to give us flexibility to take the hardware and database specific implementations away.

But I gotta say that, you know, things like Kubernetes has been just an enormous benefit and for the industry to be able to kind of move to this cloud agnostic and containerization of applications, these technologies are now mature enough that you can get pretty similar implementations across all three cloud providers. But also I gotta say, you know, the big three cloud providers are significantly more expensive than data centers. And some of the data centers are moving from bare metal to offer Kubernetes and cloud object storage.

And they're kind of like moving closer to the cloud companies, but they're not raising their prices at the same level as the Amazons and Googles of the world have. So there's a much more fluid difference between bare metal data center and fully managed cloud services. There are a lot of options in between now that are very interesting. Yeah, we run our hybrid cloud or a lab in a colo and we switched colos specifically for that reason. There are services that are undifferentiated that we don't want to manage.

Object storage is a great example of that. Like there's probably just really one object storage, open source company that we all go get our object storage from is almost always boiling down to main IO. And I don't want to manage main IO. Object storage is undifferentiated. I should just have a target that I'm writing to and I'm getting data from and I'm deleting data. It's not that complicated. Unless you're running Dropbox or something like that, you really shouldn't ever have to think about file storage infrastructure.

Exactly. And as we move up the stack, those are the types of abstractions and services that I'm advising customers to depend on. Unless there's a really great reason to manage your own storage systems, you have some type of scale that makes the math make sense. This in-between, yeah, maybe not putting it all in S3 is economical, but these kind of off-prem or near cloud solutions are giving some really great options. Yeah, and when AWS launched, they were pretty clear that there were only about five or six core services that you needed to build any application in the world.

And now they have like 200 different services. And so it's great to see some of these data center companies getting back to this, like, actually, there are only like five or six core services you really need and you need to do really well and offer them at a very affordable price. Yeah, I just sent out a note to my mailing list today that was basically challenging this notion that as we're thinking about moving workloads from the public cloud to a co-location or on-prem, whatever the situation is, we don't need to recreate the public cloud.

We made that mistake seven or eight years ago of thinking that we could recreate the public cloud. And even with all of the tooling with Kubernetes and the CNCF landscape, we could build a really great cloud, but do we really need one for the workloads that are best suited on-prem? Yeah. Have you done that calculus for your team? Kind of like, you know what? Yeah, we could build this and it would be cheaper, but. Yeah. Well, I mean, I think you've got to think about where a lot of the value is being added right now.

And, you know, object storage is so commoditized that I don't think necessarily paying significant premiums for a public cloud are necessarily the best ROI. But if you're doing something like pretty complex AI workloads, I think those are things where the public cloud are really going to shine right now, the Googles and Amazons of the world, where getting access to this type of hardware has become extremely challenging. NVIDIA has a big shortage of a lot of these chips. There's no way that you or I are going to be able to go out there and buy them.

And even if we could, you know, the amount of capital and the amortization stuff, it wouldn't make sense. But also I think, you know, a lot of the tooling and frameworks and automation that the cloud providers are offering around these LLMs and inference serving and observability, there's a huge amount of value added in these clouds in the AI space right now. Because everything's really complex still. It's also new that having a lot of it automated and handed as a managed service to you, I actually think makes a lot of sense and deserves some of the premium pricing that the clouds kind of offer for it.

But basics like file storage, I think, you know, this is a pretty competitive commoditized market. So Sean, we could easily go on for another half an hour. I have a ton of questions for you for someone who's practically doing this day in and day out. There's governance, there's employee retention. This is an extremely competitive market for this skillset. When you look at everything from your AI engineers to your platform engineers, you kind of have to be one of the world's largest search engines to justify a lot of this investment and opportunity.

There's delta between kind of your business drivers and many of the folks that listen to the show. But I do want to leave the folks with kind of how they contact and get in touch with you. And if they have search needs, how do they reach out? Sure, yeah. You can hit me up on Twitter at Sean Mulaney or on LinkedIn. I'm very responsive to both those channels or come and register on our website and, you know, hit the contact sales button.

We'll definitely reach out to you and talk to you about how to bring AI into your search box. All right, I need to hit them up and bring AI into my search box for my blog posts. They're getting insane to search now. Actually, it's not that complicated. If you want to learn more about the CTO Advisor and you want to search for other AI related content, data center related content, you can follow us on the web, the CTO Advisor. I have a nice old school search box in the right-hand corner, nothing fancy.

You can also hit me up on Twitter. DMs are open at CTO Advisor. Still use that platform. And I am extremely responsive on LinkedIn. Until then, talk to you next CTO Advisor podcast. Sean, thank you. Thank you so much.