Episode 701: Everpure’s Rob Lee on GPUs, Data Gravity, and KV Cache

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

I would look at utilization as maybe a second or third order metric to look at or maybe even an output metric. How do I optimize for getting good enough accuracy at a fast enough speed? then utilization generally is going to follow, right? you're watching or listening to Episode 701 of the CTO Advisor Podcast. I have with me a return guest, Rob Lee of Everpure, formerly Pure Storage Systems. Rob, welcome to the podcast, and I hear you got a new title.

Yeah, well, thanks for having me back, Keith. It's been a couple years now. Glad to be back. And, yeah, these days I serve as Chief Technology and Growth Officer at Everpure. So the first question I want to ask you is how does CTOs tell a compute shortage from a data problem dressed up as a compute problem? Yeah, it's an interesting question. And probably a couple of layers to that. I'd say at first blush, I would take a look at, hey, so where is all the time being spent?

Where are the resources that are being put down on the floor, whether it's GPUs, CPUs, storage, networking, what's being utilized, what's sitting idle? Generally, what's running at full bore is what you're short on in terms of resources. Generally speaking, if your GPUs are running at full bore and your storage and your network is sitting idle, you might be short on GPUs, vice versa. Now, the rub comes in on, okay, well, that's kind of a first order look at it. But how do you then know that your GPUs aren't running at full bore because you're doing a whole lot more compute, a whole lot more processing because the data you're feeding in isn't optimized well?

So that's where you have to start getting into the next level analysis of really looking at not just how much compute is being utilized, how much compute is being done in a certain amount of time on the resources you have. But is that actually useful compute? Is that the minimal set of compute you have to be doing? And I think that's where it takes a lot more, you know, kind of peeling back the covers and looking under the hood, so to speak.

So let's peel back the covers and look under the hood just a little bit. What are some of the KPIs that we should be looking at? And this is the enterprise. So, you know, this is very different than hyperscalers who are trying to get 80%, 90% utilization or higher, especially out of their GPUs. For the enterprise, we're a little bit more bursty. Batch processing, we have some real-time stuff. So we're not exactly trying to get, you know, that mystical 80% utilization.

Maybe 40% to 50% GPU utilization is on the higher end of what I see in the enterprise. What are the numbers we should be chasing? So it's interesting. I would look at utilization as maybe a second or third order metric to look at or maybe even an output metric. The primary KPI I'd be looking at is, well, maybe two in conjunction. Number one is how much time does it take to get a certain task done? And that task could be anything from a human user-driven tool, a search, a chat type experience.

It could be an agent task. It could be, you know, but for some sort of task, how long does it take for that thing to get done? Knowing that these days it's not just running it through an LLM. There's, you know, harnesses that are breaking it down into multiple steps that are doing the reasoning or farming out, you know, individual tasks and skills. But, you know, KPI number one I'd look at is, okay, so for, you know, the agent I'm building that I'm deploying to perform some business process, well, how long is that thing taking?

And, you know, how do I minimize that thing? And in conjunction with that, I would look at, okay, so what's the accuracy of that thing? And, you know, how do I optimize for getting good enough accuracy at a fast enough speed? And, you know, if you can optimize for those things, then utilization generally is going to follow, right? So said more simply, right, the faster you can do it, the less compute you're using. So I would focus on that first.

And if you can get that within acceptable means, then you kind of have a good starting point for figuring out how much infrastructure you need. Data stream puts the GPU on the storage. This, you know, gets to an interesting conversation about when we should move data and when that data should be co-located with the compute itself. This is the classic data gravity problem. How do we make that decision? Yeah. And I want to caveat this question. I need to add on a piece of it.

The reason I ask this is because we're the enterprise. And when we create a new compute stack, we create a new silo. And that's the thing that I want to avoid is creating yet another silo. I don't need a GPU storage group within my organization. Absolutely. Absolutely. And I'll caveat my answer since you got to caveat your question. With the space is moving super fast, both from a hardware perspective as well as a software and a techniques perspective.

But, you know, at the end of the day, if, you know, if we're to believe that AI driven applications, agents, what have you, AI technology is going to be pervasive in the enterprise, working on enterprise business data. And then you've got to believe that most of that processing has to happen and has to move a lot closer to the data. Right. Would be my view. You know, I think if you look back in history and when I say history, I mean like 20, 30 years in the industry.

Historically, data has been trapped within applications. Right. You pick an ISV, you pick an ERP stack, a CRM stack. You pick the application. The application figures out how to store the data. As AI becomes more prevalent, it needs to work a lot more closely with the data. It has to access the data more directly. The first step of this we've seen is, you know, in the analytics world where, you know, you start to see copies of data being moved out of applications into data lakes, into data warehouses where you can do specialized processing.

And that kind of works okay in analytics timeframes, in human timeframes. If you want to put agents to work on transactional flows, you want to put an agent to work on approving a credit card transaction, doing fraud detection, doing anomaly detection, doing security decisions. That thing's got to be transactional. It's got to be sitting in the transactional data flow. And so it's hard to see that not playing out by moving more of the compute a lot closer to the data. So we've had this discussion before, specifically around SAP.

This is something that Everpure knows exceptionally well. And as you were talking through enterprise data and this challenge with transactional data, I think automatically of SAP. SAP tried to solve the analytics problem. I think they did a good job with SAP HANA. but AI gives us this new capability that I don't think HANA was ever designed to address. As you're talking to customers who are trying to leverage that type of data, and you're looking at the shape and size of enterprise data systems today, where's the gap and where's the opportunity?

So I think there are a couple gaps, but I think the biggest gap is more structural, which is, Keith, if you look at how you frame the question, if you're a customer using SAP and you want to use AI to work with that data, in a lot of cases, that's not the question being asked, I think. I think with AI coming on the forefront, customers and enterprises, they're not looking just to work with their SAP data or their CRM data in Salesforce. They're looking to work with all of their data sets and individual silos today in unison.

Said more simply, AI now creates both the ability as well as the impetus to look at all of an enterprise's data sources, no matter where they sit, in conjunction with one another. It's no longer sufficient just to look in SAP and do an analysis and say, hey, based on the data within SAP, how would I make this decision? You've got to look at that thing. You've got to look at your CRM system. You've got to look at your support system. You've got to look at your unstructured data systems.

You've got to look at your sales contracts. And in order to do that thing, it transcends, it cuts across the boundaries of any one application stack or any one SaaS stack. Which draws us to the next problem that has become exceptionally prevalent and something that I didn't understand quite as deeply just a few months ago. And that's KV cache and KV cache offloading. When we get to the point where we have these long context windows, where we're taking all of this data from all of these multiple sources, KV cache becomes an exceptionally sticky problem in which I'm not spending money on GPUs to process data faster or the amount of data, but just to keep context in my GPUs.

How are you seeing the industry answer that problem with something like storage offloading or similar technologies? Yeah, I think the... I'm broadening your question very slightly, if you don't mind. I think it goes slightly beyond KV caching. And it's really more looking at the problem set of, hey, as you have these longer user or agent sessions, how do we avoid recomputing the same thing over and over again on the GPUs? KV cache is a piece of that. There are other pieces as well.

And I think of this as a bit like almost a new memory hierarchy. So how do you avoid recomputing lots of things? Well, it starts by having caches at multiple layers. You'll have caches on the GPUs themselves, in HBM. You've heard a lot about that in the industry today. You'll have caches in local DRAM. And then as you look at, as with any typical memory hierarchy, as you get larger and larger and further away, you'll have other tiers of that cache.

And so I think KV cache as a storage offload is becoming more and more in demand. We're seeing that a lot more from our customers, for sure, as enterprises are deploying inference in their own environments. But it doesn't stop there, right? KV cache is one type of memory, if you will. The next steps are looking at user memory or agent memory. but I think what we're going to see is that as, you know, inference and agentic systems get deployed in larger scale, but are also serving more users at the same time, the need for multiple layers of this memory will become more and more prevalent.

And remote storage is going to play a big part of that. And we're already starting to see the effects of that in the industry and in our customer base and in our portfolio. You know, Rob, you've done a pretty good job of being, staying very wide on the industry trends and not, you know, promoting Everpure. So I'm going to give you a minute here to tell me what's been announced and what's still kind of roadmappish for Everpure and how they're helping to solve some of the problems that we talked about.

Yeah, absolutely, Keith. You know, if I were to simplify it, I would say that we have a set of offers and a set of roadmap and products around AI data infrastructure, right? So how do we serve the data needs of AI through an infrastructure lens? And then we have a set of products and offers and roadmap around how do we help customers make AI data more AI ready? And the way I differentiate the two is infrastructure is about, hey, how do we make these systems run a lot faster?

How do we improve utilization? A lot of what we've spoken about so far, KV caching. How do we support the largest training environments in the world? How do we support, you know, enterprise inference? On the AI data front, we're looking at and we're solving problems like, you know, saying, hey, you know, enterprise has a ton of data. How do they organize that data in a way that makes it, you know, fast and easy to recall the right data for the right question?

Right. And the right data for the right question is really important because, you know, if you feed a ton of data into an LLM, well, that LLM has got to do a lot of work. It's inefficient. Back to utilization. It's not doing useful compute. You're spending a lot of tokens where you don't need to. And then with data intelligence, which came out of our recent acquisition of 1touch, we're really heading down the path of helping customers understand not just storing a customer's data, but helping customers understand what is that meaning?

What is the meaning of that data? Where did it come from? How should it be used? How does it correlate? How does it relate to, you know, other records and other data that they have within their enterprise? And so if you step back from it, we're covering the entire gamut from enterprises to neoclouds to hyperscalers from an AI infrastructure point of view, as well as an AI data readiness point of view. All right, Rob, I've enjoyed this conversation. com.

Until then, talk to you next episode of the CTO Advisor podcast.