Does storage matter in AI inferencing? What about the SSD?
Transcript
so gpus matter CPUs matter when we're talking about AI but what about the rest of the system components storage memory bus speeds Etc what's the relationship between Ai and we're talking mainly inferencing in the rest of your subsystems when within your data center or even within a server solid which is the spin out of tells obtain storage business unit Standalone company presented at AI Field Day 4 and they talked about the importance of the SSD so let's take this from a highlevel p perspective first what
is the importance of storage when it comes to Performance around ersing talk to the folks at Kaza who also Pres presented at AI field day and we talked about how even when we vectorize the data set when we take a set of data create a vector database so that the model can then reference your data the performance bottleneck ends up not being GPU or CPU but storage so that ability for you to retrieve the files that reference the uh question within the model I.E how fast
can a cougar go vers versus a human being in all of the test of running uh of a human running versus a human being uh human running versus a cougar all of that reference data is in a set of documents that's referenced via the vector database I can recall those documents the ability to recall those documents and deliver that to the human all of that is dependent not just on the ability of the GPU or CPU to inference and understand the vector but also retrieve the
data just plain and simple iio challenges the faster your storage subsystem the faster the response so sometimes CPU slow GPU is fine for some of your use cases you need to turn turn the knob on your storage system but solid dime isn't a storage system company not in the traditional sense they're not providing the actual sand they are either storage that goes into a system or their storage that goes into a storage subsystem you know other than size what advantages can a SSD off offer over
you know generic components from a no-name brand manufacturer and this is where solod did a okay job explaining the difference but we need to you know kind of add a little bit to that it's all about packaging why have I been a proponent for using CPUs when you can use CPUs because it's consistent with your existing operations your existing model you're not going out and building a pod dedicated for AI or a pod that can't be used for anything other than AI well what if we're
out at the edge and we want to use these Edge systems whether they have gpus or CPUs in them and we want to use them for both General compute and for AI power envelope matters the amount of ram you can uh get out of the system matters the amount of IO the amount of storage all of this changes when you're talking about the limited power envelope of a edge system in the variety of use cases you want to use this system for it is when this
low-level engineering knowing which SSD to use which mvme flash to use which CPU GPU to use within your limited power envelope which matters it is worthwhile watching the video video if you're designing systems at this level you want to learn more you can follow us on the web the futurum group uh website is futur group.com the CTO advisor the c advisor.com talk to you next CTO advisor C do
