Why RAG Breaks at Enterprise Scale — And What Comes After - Articul8
Transcript
There are no apps. In the traditional sense. Meaning the A16Z article you're talking about is written for a world where apps had to be talked through. There had to be a product manager who thinks about what it is, gives the requirements, the engineering team agonizes over it, builds something, and then it lasts over a long period of time. Today, apps last for as long as you need them to, and they go away. And they exist exactly in the moment that you need them to exist.
And they didn't exist before, they don't exist after. >> [music] >> All right, I'm super excited about this. I have a room CEO or of Articulate. We've done some pretty cool work together on just kind of this enterprise decision layer of AI, and we're extending the conversation. We're on-site at your headquarters in here in Dublin. Pretty cool room. There's a history around it. If you are and you catch me or him in the hallways at a conference, ask us about the history of this room.
It's really fun. A room, welcome to the program. Keith, thank you so much for having me. So, we have talked about this challenge of scale when it comes to vector search. RAG has been the poster child for AI. As a matter of fact, my uh cloud stack builder, 43% of the respondents say that RAG is the primary use case. But you folks are finding something in reality when when when companies go to deploy RAG. That's true. So, RAG is now become such a catch-all term for everything, and I kind of semi-jokingly say Google is a RAG.
So, doesn't mean everybody can build one, right? But the the most important paper, in my opinion, that came out just recently was the paper on semantic collapse, which basically quantified what does it take in terms of scale for a traditional RAG, which is a vector search-based RAG system, to pretty much tell you that all documents are more or less the same. The mark for that is about 10,000 documents. And the reason for that is in the vector space, when you ask a question and that gets transformed into some embedding that you're searching for, and about 10,000 documents, give or take, is when the the surface of that system that you're searching for is so large that every topic is equidistant from every other topic.
Hmm. So, that means that from a practical sense, let's say that we have a corpus of data engineering docs. 1. >> Yeah. Of a spec, and that can be Same boat with >> the results in any kind of generative AI or vector search. I mean, I don't even have to define the the results. >> Yeah. 1, right? So, it is where it is semantically related concepts, but then you maybe wanting to ask a question, but from a particular document, and you say, "I gave the name of the document.
" The problem is that document looks more or less the same as 10 other documents. I You know what's funny, you say that. I've had this exact problem with my virtual CTO advisor. I have a corpus of 500 or so docs, and I'll ask it a question about my latest two seas search, and I ask it, and it And I'll And I have to say, "I wrote on this day about this topic. " Exactly. And that's because of the semantic collapse that we're talking about.
And it's been actually reasonably well known, so nobody just implements that. You need some other grounding associated with it. But then scale is not a problem you think about after deploying the system. You have to think about it on day one. And that's really what we've been building for now over 3 years. >> So, you got to draw this out for me. Help me understand how because I'm understanding the problem, I'm not understanding the solution. Absolutely, right? And this is search is just the beginning of the problem.
Then you need to be able to to introspect, you need to be able to analyze. So, we're starting there and then building up. Okay. So, let me draw the the stack, and when we start, like let's just make sure that the infrastructure is in the bottom. And I'm calling everything the infrastructure, right? So, everything from um an enterprise wanting to run things in their own data center to running it in a public cloud, but inside their virtual private >> This is what I call the layer one, layer zero in zero my my stack.
>> Exactly. GPUs, all of the >> All of the thing where everyone access it. Everything is here. >> Okay. Now, I make a distinction after this layer, which is after you provision this, you need a data layer. Now, this could be as simple as just a database, or it could be a data warehouse. So, it could be a snowflake, it could be databricks, whatever that might be. That's that layer, right? Now, uh this is our definition, so take this with uh a bag of salt for that matter.
Uh but then on top of it is where people would draw the model layer. Now, in some cases, this is also becoming commodity. If this was just a general-purpose model, today you can go hit any one of those large models outside, and that becomes a commodity for you. It's almost like infrastructure, right? So, from an application standpoint, all of this is an infrastructure layer, right? Now, on top of this is what typically runs the agent layer. For which you need all of these.
>> Yes. And then you have the applications. Right? Now, the the difference for us is we run our model mesh and agent mesh that runs across this entire stack. So, this is what I would call agent mesh. And up to here would be model mesh. And I'll explain what that is. But if you notice, I'm explicitly pushing this down into the data data layer. >> That's That's the part that I find interesting because from this layer up, I've seen everyone play in.
But the data layer down, usually they leave it up to the snowflakes or or the databricks of the world. >> Yes. So, the reason why we operate inside the data layer is the fundamental belief and design philosophy that we have that is you do not change your system of record. You do not change move your data into a new environment. >> You're speaking to the soul of the SAP infrastructure architect. >> There you go. And it's also coming from experience of actually doing this inside enterprises where moving the data is going to be the single biggest blocker for any of these things to go to production.
Right. So, what we do is anything that we have to run on the actual data itself, as much as possible, we run on the data warehouse directly. And we only move metadata out whenever we need to. For example, the first level of information that we create to ground all of this inside the graph is what we would call a level one graph, which is just structure. Okay. Right? So, we have uh if I write that down here, um maybe I'll write it here so that you know where it is going.
So, you have um level one that is structure only. Meaning we don't know anything about what is inside the content, but it will help you solve exactly the problem you're trying to solve, which is I asked you to answer from a particular file, why are you not grounding me on the file? This knows that that file exists inside nested inside four folders sitting inside your snowflake warehouse. Right. That's all it knows. It doesn't know what's in your file. Right.
>> Right? That's level two. So, level two is content. And level three is activity. And all three of them stay inside your graph, and they get built up on as time progresses, as you are starting to interact with your content. And the difference is the reason why this has to go down here is this happens at runtime. So, when a user asks a question, you don't know anything about the content, or you know very little about the content. You may know, say, metadata about the content.
" Only when you look, you go query. And to do that effectively, and in general, most of our queries come back in less than 30 seconds, at least as a a first-level answer, you need to know exactly what to run and be as efficient as possible at the data store level. " Yes. Because RAG is not descriptive enough what's happening at this layer at the at this layer. It is this level of complexity. >> Complexity. RAG is great for small projects.
RAG is great in context of what you're trying to achieve, especially if you're filtering, you know, large pieces of data that are not similar. Totally. Uh and the other thing I'll say why you need these three levels is, first of all, if you had small volumes of data, sure, you can process all of it at once. But in most enterprises, the volume of data is there, but the usefulness of that content is less than 10% today. Now, you can use more if you knew what is in your data set, but you don't.
Like most of the data sets most enterprises use maybe 10%. I'm being actually very optimistic about it. It's way less than 5%. That's why you don't want to process everything. So, you're you're you're pulling the data scientist out of the a little bit. A couple of weeks ago, I took all of the TechFieldDay videos. 7,000 TechFieldDay 7 hours of video. And I said, "You know what? I should make the acts intelligent. This is the the days of GenAI.
" I spent almost all of my time needing to classify what divide, classify, and label the data. And even that classification, once I was done with it, I can't just feed that to a model. It's still too much data. >> Yes. But, even worse, right? Think about it. We are in the GenAI era where things are supposed to happen automatically. And here we are sitting and tagging because we know that without that you won't even get basic answers.
This is the tagging that has to happen automatically. That's really why building these graphs are non-trivial. You need to be able to decipher that. I'll give you a like a seemingly simple problem, right? So, say you're on Azure. Mhm. You have say a few thousand documents in Azure blob store. Now, Azure blob store is flat. It may look like you have folders, but then when you're programmatically asking for information, they're all flat. They're all objects. >> Right.
If you ask a question simple question of how many folders are in my blog, not a trivial thing to answer. Because you have to go parse everything, you have to go and give the answers. And when you're building into enterprise systems, you can't take these things for granted. And if your integration becomes so onerous, the application that is supposed to give you your ROI now became 10 times more expensive. So, you're I was comparing kind of this approach to the A16Z article that we've read in before.
And they start here. They start at the app and the app controls everything underneath. They And that's great if you're single app. Yes. But, if you're a enterprise trying to provide this as a service to apps, that becomes almost unmanageable. >> Yes. And that's what it is today. The reason why you see only 5% of these projects is going to production is exactly that. Because you can get to a small sliver and maybe one or two applications.
Scaling that is not easy. That's one and it's almost become impossible. The second thing is I have a maybe a a slightly controversial statement to make, which is there are no apps in the traditional sense. Meaning the A16Z article you're talking about is written for a world where apps had to be thought through. There had to be a product manager who thinks about what it is, gives the requirements, the engineering team agonizes over it, builds something, and then it lasts for a long period of time.
Today, apps last for as long as you need them to and they go away. And they exist exactly in the moment that you need them to exist. And they didn't exist before, they don't exist after. But, I'm intrigued at how this now becomes a contract. The agent and models now become a contract as part of the agent mesh and the model mesh. So, when I'm building my business capability on top of my data set, my infrastructure, my data, my model, my agents, and my what we are calling apps, I need that to be a consistent contract.
That's right. So, I will qualify that as there's no static apps. Now, it's uh what I would call them as real-time apps. Which is the apps become available when you need them to. Meaning including the agents. So, in the agent mesh we have this thing called missions. That if you want to do something, for example, if you want to analyze um all of the current um quarterly reports from Tesla and you want to look at what the market is doing in the automotive sector in the EV world and come up with a thesis for what is the risk of investing in Tesla today, not like say this quarter or anything today based on what the market dynamics is doing.
That's the complicated mission you define. And then what agents in what combination of agents have to be used to satisfy your mission gets generated on the fly. Yeah, I'm wishing that I would have engaged you folks before I undertook the dividing 7,000 videos into 230,000 segments, then categorizing the segments, tagging the data, and then all to realize that if I wanted to look at the data from another perspective, they would have had to redo the whole thing. >> Exactly. This solution seems to solve that problem for me.
I have the data lake and now just need a graph that's on top of that data lake and now I can the ask the the different questions. >> Exactly. And all of these missions are grounded on this graph. Like and the missions basically are comprised of what question you asked, which is a combination of squads, which is just a collection of agents. And agents, but the thing that we haven't talked about here is agents is just a combination of models and tools.
So, we're writing this as if it's simple. I mean, I can tell you from experience, yes, this is not easy. That's true. That is true. So, to build a single agent today, it's okay, you can craft it. Actually collecting a group of agents and then getting them to do something for you is incredibly hard. In fact, what we've realized is even the largest players out there have anywhere between 10 to 15 retries per agent tool call. You just don't see it because behind the scenes it happens so fast.
>> Right. But, these agents are extremely fragile if you don't ground them properly. That's why you need the graph. That's why you need something called an agent mesh that does it for you. And in an enterprise setting, if you got one answer with three retries and another answer with 10 retries in an audit trail, I need to be able to say that. Right. Today, almost none of these systems will like tell you how to do the audit on that one.
That's really how they do a cost audit, they won't do a quality audit. And you know, I just wrote framework or a model on decision authority placement and this idea that if I'm giving AI or some automated agent a authority to make a decision on my behalf, I need to know the impact of that and what and and I need to be able to audit it. So, you're talking to that example specifically. And this is not just for regulated industry. You need to understand and you know, we we we've called it uh explainable AI.
Uh but the basic we can't give a black box uh the keys to our business without understanding what it's doing. >> Absolutely, right? And the other thing on top of missions is what we call digital twins. And the terminology here is very deliberate. This is not a marketing jargon. The reason why it is very deliberately done this way is if you notice it's like a Russian doll one building on top of the other. You have data, models are built on data.
Agents have tools in between, so that's where the tooling layer comes in. So, tools plus data plus models become agents. A group of agents become squads to deliver a particular mission for you. So, all of this can be ephemeral. At that point in time they're built up and then they go away. The building blocks are models, tools, and data. Sometimes an agent can also be hardened saying I absolutely want to use the same agent again and again. That's what it gets hardened on.
Missions come and go. Digital twins are a combination of missions. So, what I mean by that is if you say the Tesla example I used, you can say there is a digital twin of the market that you care about. It's not the entire market, but the market you care about. Maybe it's automotive, but it's tier two suppliers in the automotive sector. You create a digital twin of that to constantly be updated based on what is happening in the world. And you also add simulation tools in it.
So, these tools are what LLMs can call. This can be even more heavy duty. So, this has simulations in it. Meaning you can say, "Okay, I have this set of models, this set of mission to run. " So, answer scenarios. So, we're using traditional investment as a uh model, but this happens in any other domain. >> Any other domain. Oil and gas, it happens I I was talking to a oil and gas CTO uh uh yes the other day.
Actually, they were reacting to your paper. " But, also in traditional like geek stuff like IT infrastructure, what happens when I invest in this set of network capability, which is completely like fungible for for this type of scenario. >> So, in fact, we launched agents when we launched our agents in the AWS marketplace as a launch partner. One of the agents we launched was called VIV. It's still available. It does network simulations and network anomaly detection. That feeds into a digital twin of network analysis where all you do is feed in a bunch of say logs and a bunch of reports and it generates the topology of the network without knowing the topology.
And that is one of the toughest problems in networking, right? We can't do AB testing in networking because the possibilities are too >> Too much. Too much. And here, because we had the graph to ground on, we just use the same technology to get the topology. Same thing in design. You mentioned oil and gas, right? So, depending on whether you are in upstream or downstream in oil and gas, you have massive amounts of time series information. You have lots of traditional tools that you use already to predict what is going to happen.
Now, this turbocharges it. Right? Where you not only get an LLM to work with your traditional tool, you have a system to actually call your tools when you need it. Aram, this has been a fascinating conversation. I still have a ton of questions. >> Absolutely. I'll direct folks to the Tech Field Day video that you did a little while ago. But, uh I'm this was a ball. Did we miss anything? So, the other thing we missed, in addition to scale, is accuracy.
Because we started a little bit on accuracy, where, okay, you can say get to your specific document from a specific uh say file name using say keyword grounding and things like that, right? What happens if you have hundreds of thousands of keywords? What happens if you have not only keywords to worry about, but you also want the keywords to be similar? So, you mentioned oil and gas. Oil and gas, depending on which jurisdiction you act in, has different units that you need to adhere to.
The acronyms that exist inside all enterprises. Each enterprise has an acronym soup that you need to be intimately familiar with. When you're asking questions about analyzing information, you need to be familiar with all of it. That is the real killer, because you need accuracy first before you get to scale. Yeah, because for my business user, if it doesn't understand my language, what what use is of it? If I have learn another language, and I'm not talking about a programming language, but another taxonomy, taxonomy of discussing, this is why broad tools sometimes don't work in within specific industries.
>> Exactly. Yeah. Aram, this has been an amazing conversation. We'll put some links in the video session below. If you want to learn more about Articulate, it is as it's presented on screen. Pretty cool eight logo inside of that. com. I'm as always the CTO Advisor on the web @CTOAdvisor on social media. Perfect. Keith, thank you so much. Thanks a lot. Thanks a lot. Thank you.
