AI Inference on a Dell Laptop
Transcript
(gentle music) >> Welcome to A CTO Advisor video. (graphic swoosh) I'm Alastair Cooke, and in this video we're working with Dell to take a look at some generative AI. Let's step back for a moment and set the scene. For the last year or so, it's been impossible to avoid hearing about large language models. ChatGPT broke onto the scene and we've seen other large language models being built, and you could ask them questions. Now there's a couple of restrictions on this, because the actual trawl of information taken from the internet is at a fixed time, and there is a massive cost to training these models.
Sam Altman said that it cost up to a hundred million dollars to train one of the GPT models. And so you can imagine that there aren't many organizations that want to spend that money. For example, this is a PDF document, one of the PDF documents we're actually using in the demo that has a study of accelerating storage performance. And this document may have been released after the LLaMA 2 model that we are using was trained. And there's essentially two ways that we can take these large language models that have cost millions, tens, hundreds of millions of dollars to train and bring them in with our knowledge.
The two techniques are the retrieval method that we're studying in this particular video. And the other technique is called fine tuning. Now, fine tuning is a continual training. Fine tuning adds training on a specific knowledge set to an existing large language model. And so what it does is it produces a newer large language model with that newer information. And so fine tuning is really good if you have a private set of data that's not changing so often, but that training's really intensive and therefore expensive.
And this is why we get to that retrieval side, where rather than modifying the large language model, we supplement it. And we supplement it with a thing called a vector database. We take our domain specific potentially private information, we put it through the split and encode process to put it into a vector database. And then we have this library in the middle here called LangChain that's used to combine together information from that vector database. And then pull information out of that vector database based on our user input prompting, and then feed this collection of augmented data out to the large language model to get a response from the large language model.
I'm running that generative AI actually on a laptop right here in front of us. So this isn't any old laptop, this is a beautiful Dell Precision 7780 laptop. It's got an RTX 500 card in it, that's an Nvidia graphics acceleration card. And it's that Nvidia card that is the special source that makes this a great laptop for running these large language models that require that massive parallel compute that GPUs are known for. This laptop also has lots of memory and lots of CPU power, 24 cores.
This is a beast. I'd like to take it home with me. But what we're going to do is take a look at how you might use a large language model and then augment its ability to understand what you're asking it with a bit newer knowledge. And this is a technique known as retrieval augmented generation or RAG, where you take the trained large language model and then you vectorize some information. That's a way of saying, you take in some body of information.
For the demo that I'll show you, that body of information is about 790 documents from the Dell PowerEdge server platforms document set. I'm going to ask that large language model questions about that data set, and this retrieval augmented generation methodology is very much about enriching a large language model with some domain specific and up-to-date knowledge. We'll start off with a large language model that really started the whole hype around large language models, which is ChatGPT. And I'm going to ask ChatGPT a fairly basic question around deploying on PowerEdge servers.
0 U2? Not quite the latest, but not the oldest. And here we have one of the challenges with the large language model. ChatGPT, this GPT-4 engine was trained in January, 2022. 0 U2 didn't exist at the time, so it doesn't know anything about it. Nicely, ChatGPT didn't just make up an answer with a hallucination based on what the previous answers might have been for things it didn't know about, and it did give some nice generic guidance about go and read some documentation.
7? That was definitely around. Excellent. We got an answer, we know something, but the guidance here is go read the documentation. Well, I knew to read the documentation before, so that's not really very helpful guidance. And this is why we like, there's two factors, right? The up-to-date information and specific information is why we move towards these retrieval augmented generation where we're providing this body of knowledge that we want to add. And that's what I've got in this other window here.
So this is running entirely on this machine. So we're seeing a large language model on a relatively modest Nvidia GPU. And we can ask the same questions. In fact, let's copy and paste the questions. 0, we'll paste that in and ask the question on our local large language model. And this is LLaMA 2 running locally. And then there's that additional information from those 791 documents. And even though I didn't type the, you know, get the W in there, there's some recommendations in here.
So there is some specific recommendations around installing, says there is absolutely a specific requirements. 60 or greater is recommended in the ESXi update guide. There's some more recommendations here around that other models of servers as well. One of the funny things about large language models is that if you phrase your question slightly differently, you get maybe a slightly different answer. Last time, I actually got much more direct answers on this one. 7? Can we get something more direct on that one?
And of course we've got context, we've got more information sitting behind. So yes, the BIOS versions are different. 0 for that R740F, that happens to have been what the large language model has chosen to use. Now I've asked the sorts of questions I'd be asking during a design activity, and where I'm putting together an upgrade plan potentially to go from one version to another on top of these servers in our data center. This particular demonstration, that has been put together by the Dell team, has some other prompts of things that you might want to ask.
And you can see that a lot of them are much less detailed, technical, and things like, give me a CTO advisory proposal, cluster aware upgrade in VxRail. Yeah, tell me about that process. So there's a lot of answers being given here about the content of those documents that we provided. And although this is specific to the Dell server infrastructure, this is also applicable in other places. So anywhere there's this big body of data and particularly a body of data that's being updated over time, you can use this retrieval augmented generation in order to update the information the large language model has.
This is a much lower load than actually updating the entire LLM itself. So whilst these large language models takes weeks and months to update, the retrieval augmented generation, the RAG component, I updated on this laptop in a matter of a few minutes. And so if there's a growing body of data that you want to have feeding this LLM, this methodology is really good. It doesn't require a huge amount of resource. And whilst this laptop is great, when it's me wanting to do the work or if I want to develop some things or if I need to use it disconnected, there's some scalability constraints around this.
We probably want to use a shared resource. The GPU is idle while I'm sitting here talking to you, or if I'm doing my emails, maybe it would be better to be running this large language model in a shared resource like your own data center, on top of the existing Dell servers that you have or new Dell servers with the Nvidia cards in them. You can see that as being a shared pooled resource to provide this large language model. But also you get the benefit that all that domain specific knowledge, all of the files that represent your business' knowledge, in this case, these 791 PDFs, but in your case probably hundreds and thousands of documents that have been generated over time in the organization, they remain inside your data center.
And so they remain in your control, they're not leaking out anywhere. Thank you for joining me on this exploration of running retrieval augmented generation on relatively low power devices. You can see that we could be running this in data centers. Now to catch up with more of this, Nvidia GTC is one of the conferences that you'll find Dell Technologies in the CTO Advisor attending. If you're catching up with this video after GTC this year, well, Dell Technologies World is upcoming as well and I expect to see quite a lot of AI innovation from Dell at Dell Technologies World.
And you should see the CTO Advisor also at Dell Technologies World. Do keep track of what we're producing. There's a really interesting area of making sure that you're getting the most business value out of these breakthrough technologies. Stay tuned to the CTO Advisor and to Dell Technologies.
