10M Token Context Windows vs RAG: Do You Still Need Retrieval?
Transcript
All right. 0 has a 10 million token context window, we no longer need RAD. Let's put some stuff into perspective. These first, these two things are not the same. We'll get into that, but let's talk about the accomplishment of a 10 million token context window. That means I can put the equivalent of roughly 40 mega I'm sorry. Yeah, 40 megabytes of data in a context in a single cury or context window. So in a prompt, how big is 40 megabytes?
That's roughly 80 novels. That's a lot of data. So, I could take an entire Lord of the Rings Return of the King novel and paste it into my GPT prompt and ask questions of like, hey, when did Fert and Sam reach Mordor, if at all, in this book? And without needing to read the novel, that prompt is sent to the LLM and it processes and then I get the answer. Hey, okay, now I don't need Rag anymore, right? Not so simple.
Again, these things are not the same. rag. If I was to vectorize the novel, put embed it into my LLM and now my LLM can process it as I ask questions. I don't need to load the entire roughly 125,000 tokens into memory every time you ask a question. So, every time I ask a question in the context window example, I'm charged for all 125,000 tokens every time I prompt using the large context window. When I do it with rag, the backend process is just make it simpler because I'm not inferencing off the entire data set.
I'm using the optimization of my AI platform. Secondarily, even if I wanted to do that, this is not efficient. I'm giving the LLM too much noise and I'm working against myself. The purpose of a context window is to give again context to your search or to your prompt so that you're giving the LLM pertinent data to what you're doing. So let's think about a large application that you're developing and you want to give it several files or several uh uh long lines of code so that you know what APA API calls are being made etc etc and it can use this large language model in the background around that language to understand your request.
That's just the simple differences between that. There's also the problem of scale. 40 megabytes of data, which is the equivalent of roughly 10 million tokens, is not that much data when you're talking about enterprise data. If you have a 300 gigabyte SharePoint library that you want to inference against, that 10 million token context window is kind of useless. So, I hope that helps.
