Borrow the Vendor's Plumbing, Not Its Judgment | RAG on AWS + NVIDIA DGX Spark
Transcript
This video we touch on the economics of running inferencing on a NVIDIA Spark versus the economics of running inference in AWS. But this isn't a comparison of the Spark versus AWS. That's not a interesting question. This it's been answered. It is about it is about where I see AI projects fail the most in practice. And this is when we loan judgment or borrow judgment from the public clouds and our AI pipeline without having placed that authority specifically in these entities.
So what did I find out in this lab? Let's talk about what we built. 13 years of video transcripts about 237,000 segments. The data pipe line is we've uploaded all of those chunks into AWS S3 in S3 vectors. The box is Nvidia DGX Spark as mentioned GBTN sitting right here next to me right now and four plus one terms layer zero compute the Nvidia Spark layer one the data plane living in AWS then layer two in reasoning again back here on the Spark a clean partition each vendor owning whole layers not subdividing the layers.
I walked in asking where does the bottleneck fail. I walked with two different questions. Walked away with two different questions. Let's again be clear out of the gate. The economics is not even a comparison and it is not what's up for debate. It took 84 cent for me to seed this project into AWS S3 and to create basically the rack pipeline to 20 cent of inferencing once I got the data up there and compared the inferencing to the spark. The spark costs $4,700.
Now you'd have to do a awful lot of inferencing. Matter of fact, we found that there's no amount of inferencing that you can do on the Spark to justify this at a price point, mainly because of throughput. AWS is 7 to nine times faster running the same model. But again, this isn't the major finding. The major finding was in building the components within AWS. The initial approach was to take bedrock and create a knowledge base and then I feed the spark based on that knowledge b uh base rapper.
What I discovered is that similar to other cloud platforms it takes away a lot of the decisions I need to own. I'm creating a production rag system. uh it rechunks into its own strategy which I don't have the knob to turn. Uh it parses on things I didn't ask for and it uh broke labels as a result. So the first learning from this lab is bring your own primitives one segment one vector labels attached for this project a hybrid filtering model and then keep your chunking decisions.
com where you can understand these tradeoffs. So the rapper again the strength of the primitives and not the rapper. What I learned was because of I you can call it uh operator error or uh Jeff operator luck I could not deploy the knowledge base programmatically through the CLI. So I had to go and do it uh directly. Uh that was too much of a pain. So what I decided to do was to build the uh the the capability from the primitive Titan to embeddings and then uh AWS vector uh S3 directly.
You can go back and read the lab notes. This revealed this exceptionally tunable capability that I had not uh considered as part of my initial uh design. So from a primitives perspective, cheap, fast, hybrid, all the capabilities that I wanted and more control in AWS. AWS proved to be exceptionally capable in doing this. And the ultimate point of all of my layer 2C or 4 plus one AI infrastructure research is to keep control of your reasoning. And I'm not talking about thinking models when it comes to reasoning.
I'm talking about judgment. The whole point of this exercise and what I discovered when I leave chunking over to AWS, I was losing some of my judgment or reasoning capability because I outsourced that decision making to AWS. AWS controlled that gate and I would have had to create all kind of hacks on the back end to reclaim that judgment. This is the one thing that this project taught me against my initial 4 plus one AI infrastructure assessment against AWS and I kind of set out bedrock as the bedrock knowledge base as the example rag pipeline that's not it's the primitives itself that AWS offered that was the value and by me controlling that structure I controlled the reasoning plane of this project which was to again get insights from my data.
So where did authority actually sit once we sat down from layer to layer? Layer one which is storage object storage was seated to AWS. AWS controls the infrastructure. They control uh everything around S3. I can configure it but that responsibility is seated to AWS retrieval. So the vectors uh store the rack pipeline what I was hoping that bedrock would do for me. Again all those primitives are AWS's and they're seated to me. What did I retain myself the underlying compute I could switch out my Nvidia Spark for AMD system.
I can switch it out for a Dell system. I control that. Same thing with the runtime. And again, we talked about the reasoning. Chunking isn't reasoning. The data and the business process I want to get out of it is the reasoning. And I retained all of that myself on premises. com I kind of compared vast datab bricks AWS all in the same bucket and what actually going through the motions taught me was that there is a floor that AWS establishes that's a pretty high bar.
It took me about 41 minutes to ingest all 300 or 237,000 uh segments. That's pretty robust. And the I didn't find that the latency in cury was a problem in my application. So what value or when does the specialist data vis bricks vast when do they come into play? Interesting question. And then there's the unanswered math problem that I uncovered around when does onprim beat public cloud when it comes to infraine. There is a scenario specifically I discovered it in this test.
It was when the model is a fine-tuned model the pricing floor changes. AWS no longer charges me per token, but there's a uh there's there's an entry fee to even have a fine tune model. What is that entry fee and how does that compare to just buying your own local compute? So, in conclusion, we have to be honest about what we didn't prove. I didn't prove that on pay on prim never pays. That's a to be determined project. I didn't settle custom model economics.
We're going to get into that. It didn't prove manage rag is bad. We just saw where some of the potholes exist. These products exist for a reason. Future labs will explore all of these aspects. If you have a question, if there's a a aspect of the lab I didn't mention and you're watching this on YouTube, comment below. What What would you like to see me go with it? What didn't I answer? What didn't the lab notes of this project not inform you of?
Want to learn more? com. com. You can follow me on most social platforms at CTO Advisor. Thanks a lot.
