How AI Turned 80 Years of “Dirty Data” into a Smart City Platform
Transcript
We helped them tackle their dirty data when it came to deep management. They have literally thousands of these deeds on microfiche. >> [music] >> All right. I'm excited. We are in the town of Vail. As you can see from the B-roll and the beautiful backdrop. And I have with us the sponsor of this video, Robin Bond, VP of AI and business development, HP. >> Yeah. >> And if you're familiar with the channel, you've seen Luke Noresize a few times.
Ironically, I think we've created more content in Colorado than anywhere else. From Granby to all kinds of beautiful parts. This is my first time in Vail. Beautiful, beautiful town. But we're not here just to talk ski slopes or the amount of snow that fell. You folks did an amazing project with the town of Vail this year. Quick overview, what Luke >> We're doing an amazing project. It's ongoing and it's still going. >> I I say did because you presented at Tech Field Day a few months ago about the project.
Get >> a bunch of questions. The delegates were very impressed with kind of what you had done up to that point. But what is the project? >> So it's multifaceted. It started off with four discrete use cases. The one that I'm most excited about, and I'll I'll let Robin explain the others, was we helped them tackle their dirty data when it came to deep management. They have literally thousands of these deeds on microfiche. So I love it if you're under 40, microfish >> [laughter] >> is probably I was explaining earlier to you.
Yeah, exactly. It's effectively they scanned a lot of old paper documents and put it on to a narrow film that you can then store that one copy of that narrow film in like a handful and it might have, you know, thousands if not tens of thousands of documents. And now they've also digitized just that microfish. So, what was amazing about it is these are documents back to like the 40s. Um uh in some cases handwritten stamped deeds and deed restrictions all throughout this town because the land is literally some of the most expensive land, let alone the rental and sort of purchase properties.
So, what we're able to do is help get AI to actually read all of that data, put that all into an anthology, and allow a small group of individuals now to manage the entire deed flow process, net new deeds, deed restrictions, deed compliancy, and a sort of one-stop AI-driven shop. And it's been a real game-changer for them to show how they can move these arcane processes of running a town into sort of the future in AI-driven. >> So, Robin, I actually have a couple of questions from the HPE side, but Luke, you're striking kind of this this sweet spot in the industry where you're talking about ontology of the data.
Project after project after project, I've seen this repeated problem that hallucination happens even when I point my large language model, whether it's on a private infrastructure hosted in a public cloud provider, really doesn't matter, I still get hallucination. It's just hallucination around the data. " And I still it still won't return the results. >> Yep, yep, that's an actual thing. So, what we've seen over the last 3 to 6 months is the idea of retrieval argument generation has really changed. And what I mean by that is we first build those embeddings of all of the documents.
We put those embeddings into a vector database. And then from there we build a metadata structure on how to access that data and then we build an auto anthology from the ground up of what the data is, the entities and the abstractions of it. I eat through each lens the same word can mean something different. And then when we inference, we inference right back down to that stack. " And now we retrieve the whole documents through the vector database, pull in the whole documents.
Now it actually infers the documents the anthology together. And keep in mind the inference is now just logic. It is actually no longer trying to infer an answer. It's trying to infer the logic on how to go through an anthology. So, your hallucinations now go to near zero if you have a thinking model cuz it lays out here's how I'm going to process this anthology and then it just executes that. And it's actually been a game changer in the enterprise.
>> So, I get this concept across it really doesn't matter where a model is being served. I need to put this reasoning level layer between my data, my raw data that's it vectorized, and my large language models and I think Robin, this is where HPE comes in. When I first heard about this project, my curiosity was who's tackling large language models in a private data center setting when you can simply outsource the large language model to other uh providers a lot of the other cloud providers.
What was the business case for Vail to bring this to to type of private data center? >> Well, I think there's two pieces and I love kind of the use case that Luke started with. Uh but the whole idea around bringing together multiple different types of AI with kind of Commvault as that identical platform and kind of glue behind it to break down those silos lets us really see the impact of starting to connect and kind of where you're going where you have different types of AI, you start to bring that in and then to Luke's point you start to be able whether it's with vision AI or with this kind of this work that we've been doing down on kind of that dirty data with deeds that you start to be able to do this inferencing and you start to be able to do that real lean in with that reasoning, that thinking um and being able to tie into that ontology.
Now, part of the reason Now, there's two reasons to do to bring that um within the private data center. And and I love Veeam's data center. It's like all off of sustainable energy. Um I mean, they're they're really have done just an amazing They really leaned in and and are doing an amazing job of of how they're they're walking the walk of of being um responsible um to to this area. Uh but anyways, the um but being able to to bring that within their data center it allows them to leverage all of this data without having to share it out publicly um so that it can kind of be done safely within their confines uh as well as when you start to look at the cost of doing some of this repetitively in the cloud is like the economics just don't make sense.
And and so when we can do that in a way and you can kind of control that token economics better within the private data center within the or within the private cloud as well as being able to but still like work with Commvault as on our other partners being able to bring all of that intelligence but but here and then to be able to do it within their security confines so that they're not having to open up their data kind of outside of their firewall just to bring intelligence to it.
I love bringing the intelligence to the data rather than the data to the intelligence. >> Yeah, and moving data is not something that I suggest. I don't care how powerful the AI system is, eventually data gravity wins. It always wins. And Luke, Robin just mentioned something that me and you have talked about in the past. One of the rationales that I get from the fans of the big models is that just make the model bigger, make the model better, and that will solve your data problem.
But that solves that actually creates two different problems. One, it gets really expensive >> to do really fast. >> uh data cleanup or data rationalization cuz you have to consume more tokens. And the other thing is sometimes the big models are overkill or not really appropriate for the task at hand. Sometimes you need a specialized model. Talk to me about the flexibility Kamiwaza brings for using multiple models to get the work done. >> Well, I mean, just in the workflows here at Val, it's typically three to four models to actually get an entire workflow.
Uh once again, you start with those large sort of reasoning models, and these are still, you know, private level uh reasoning models that break out the tasks, break out the effort, and break out the individual things that need to be done. Then those pass to more discrete models that are very focused on processing, whether it's writing code for a repeatable process, document imaging, etc. So, I mean, you get three to four of those models together, you got to orchestrate that, and that becomes incredibly powerful.
Where if you had one model that had uh uh token limits, that had token context limits, you literally couldn't accomplish that. Um so, it it it's it's a misnomer to say just a larger model with larger context is going to be able to do it. Second, um one of the use cases is fire detection. We're talking hundreds of cameras all throughout here, scanning the mountainside, scanning everything. That means every second you're having a model crunch the data of what that actual screen looks like.
You're talking trillions of tokens probably in a week. That is not something you literally could move the data just because it's streaming so much data from all the cameras into their central data center to be processed. I don't even know how you would be able to move that out of this location. And if you could, we're not talking thousands, we're talking 10,000s or millions of dollars of bandwidth that you're estimating gross charges in and out of the cloud. It would just be egregious.
>> So, look, you're you're hitting a popular tone when it comes to private AI, but scale breaks everything. And you just brought up an example of scale, not necessarily model scale, but deployment scale. When I have thousands of or tens of thousands of nodes that I have to deploy a platform to in order to update code, you know, getting the latest model to those fire detection units is literally the difference between life and death or property loss. I think that's where, you know, we start to get scared.
Like, how how how am I going to manage the control plane? So, how is HPE and Comiwaise helped to solve this scale problem for the town of Vail? >> Um so, Dust start that HPE with the PCIA platform. The fact that they have an orchestration capability for all the hardware, for all the maintenance of that hardware, for all the code base, the drivers, etc. is sort of a game changer to build on top of. Because that means one or two admins here in the town of Vail can manage a fairly decent size cluster and scalability.
You layer us on top of that to manage the scalability in the direction, the orchestration of the platform. e. each one of those inference requests going to the right card that has the right context for the right model. Bam, now you sort of have a dual sort of approach on managing the AI orchestration of the infrastructure. >> So, Rob, and this sounds familiar. We just did a paper talking about the power of reasoning at the infrastructure level and how Vail has deployed this level of reasoning this is going to layer deeper.
Talk to me about the orchestration from we this is the logical the physical. One of the things that customers are definitely afraid of is just managing the size of the equipment uh commitment. This is another area of scale. How is HPE helping Vail manage the life cycle the refresh? Uh AI is moving faster than from a technology perspective that anyone can keep up with even on the interesting side. How are you hoping to tell them to manage this? >> Uh that's a great question because you know the speed of AI is is this just this through the roof and then so how do you have how do you help folks like Vail manage that?
And and to the point of what Luke was referencing it's starting with like our private cloud for AI which allows us it's purpose-built for inferencing but it's purpose-built with an integrated AI stack. What that allows is that that means it's life cycle managed for them throughout the life cycle of that of that platform. And so they don't have to be going out and suddenly become Kubernetes experts and managing that or being experts on how to manage a GPU because all of that kind of orchestration and logic is built into the platform as a starting point and will continue to be managed will continue to be expanded and then it's easy to expand and um and of course we in HPE we do love talking about GreenLake of however the real reason that we love that in this type of instance um is that PCAI allows you to continue to expand to bring in like expansion modules as you continue to build out use cases with Commvault and other AI partners as part of the smart city approach is you're going to be like oh but wait we want to do insert your new favorite use case here.
You know, maybe you do need more more hardware to support that. Well, we can easily drop in an expansion rack. It's automatically um you know, kind of included in the PCAI, so now that's also life cycle managed as part of it, but we make that really easy with GreenLake, so that you can kind of pay as that kind of subscription over time and just be able to kind of have a change order and add things as you need to kind of in that cloud-like experience, but all within their data center.
So, I think those are the things and we're also looking at ways to build in that technology refresh, those life cycle management pieces of that underlying technology to go with the to go with the software, to go with the um kind of logical capabilities on top, cuz we're going to, you know, continue to innovate at you know, we're going to be running right behind Nvidia as they innovate. >> So, last area of kind of thick foot that I like to talk about is developer experience.
Look, the you know, you followed me on I you've followed my AI journey from being a AI newbie to being a AI newbie plus one. And I had this experience of trying to get VLM loaded on my Nvidia Spark. It is, you know, purpose designed Mhm. to run this and I'm like, this is supposed to save me from uh driver hell and it hasn't. I think one of the the I think barriers to private AI has been that fear of giving developers this cloud-like experience where they're just dealing with APIs and not developers and make it simple.
I'll challenge the statement that only a couple of admins can run the whole infrastructure. It takes me 3 weeks that just to get a driver working on bare metal hardware. How's Kamiwaza overcoming that operational challenge. >> So, that is literally what we take care of. So, whether As soon as you install Kamiwaza, and even that now is down to about one command, pull down all the repos and get you going. Then we interrogate the hardware, which is real simple cuz we have an API call right into the PCAI, which then tells us what the hardware is, and then we'll redeploy the actual inference engine optimized to the underlying hardware.
Once all of that's established, you then just point us to the directories, the object storage, your primary databases, and then from there we provide an API and an SDK. So, now developer literally, if they're using Cloud or Codex, they can tell the applications they want, they point it to our SDK, they say make it deployable on that, and now it's a one push, literally one button push. >> Well, I really appreciate you two taking the time to come over the mountain cuz both of you had to come over a mountain and the pass, over the pass, through the pass.
And it's worth it. This was a conversation that I've been looking forward to. I really wanted to challenge both of you on kind of the value of not just uh private AI, but private AI's on HPE and a startup in a town that is I think it's safe to say Yuba Valley is conservative. Yeah. Using a startup on top of HPE to deploy mission-critical workloads on smart city setting. I really appreciate you two taking the time. You want to learn more about the CTO Advisor, you can follow us on the web.
Links to both Kamiwaza and HPE's AI efforts will be below. And our the white paper we did on the project. >> Great paper. >> Yeah, links were below that.
