Cloud Identity Management with Hashicorp Vault & vSphere

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

Hey, you're listening to another episode of the CTO. As I've been told, that's my official intro. So just to make everyone comfortable, I'm going to stay with that. And speaking of comfortable, we have a returning guest, Martez Reed. Marketing director for Morpheus Data. Martez, welcome back. Thanks for having me again, Keith. You know what? One of these days we're going to do a podcast in person. Like literally, you know, speaking in Chicago is nothing's really right down the street, but we're in the same region.

And at some point we need to just, you know, jump into the same room. But working from home is so, so nice. Yeah, it's definitely convenient. I don't miss the going downtown, spending two hours trying to get back home. No, I do not miss that. And if you're not familiar with the Chicagoland area, I don't unless you're living like if you have to work downtown and yet in your. You have to commute into work unless you're unless you live downtown.

The commute is a minimum a half an hour to two hours for a distance that's not very far. Maybe 25, maybe 25 miles. Most people commute in and it's a brutal, brutal commute. So, Martez, you posted a tweet that you combined two technologies to make a solution that teased me a little bit. And I'm thinking, OK, that's that's really interesting. So I'm going to give you the way that I understand it. You can correct me if I'm wrong. And then we're going to go into kind of the details of what your vision is for the project that you're working on.

So let's start out with a problem statement. The problem statement is that when you look at modern cloud capabilities, specifically AWS, AWS is IAM is amazing. Like what you can do, the power of the sensibility of IAM. It's not just about securing systems. It's about providing access to. Applications and roles via identity that we could never do as easily as we can do today. And I am so an example that a AWS cloud engineer gave me was to take a easy to.

Instance, give it a role it now has identifier in this easy to environment and want customers VPC. I can then assign it to a different security object in another environment. And in another customer environment and without doing federation, et cetera, the the the universal nature of IAM objects allows me to assign a role or rights to any other object, no matter the account. Demark is super, super powerful. And you posted something to the effect of taking HashiCorp's ball. Giving a.

Identity object identity to a VM like a VM where VM and similar capabilities is, is that the basic high level overview what you were talking about? Yeah, I mean, at the gist of it, I mean, even even boiling it down even simpler. When you start talking about single AWS account, I want a resource, a VM, easy to instance to be able to access, let's say, an RDS database. How do I give that VM access to access the RDS database or whether it be pushed logs into CloudWatch?

Typically, that's done via IAM role. And so I associate some form of identity with the easy to instance. And then I can start to easily tie it into other services, cloud native services within that particular cloud. It makes it extremely simple and easy to start looking at ways to to scale those application in terms of deployment across the environment without having to worry about, OK, credential here, credential there. And instead, we have to go through the challenge of keeping up with credentials across application deployments.

Yeah, it makes it when you think about application, not just application deployments. As you develop a microservices based architect architecture for your apps, whether those microservices are VM based or you using native cloud services or if you're building the applications or the services yourselves to not repeat credentials across multiple apps. This this this trusted ecosystem makes for a much, much more robust application development process than if I had to manage secrets across multiple apps, multiple customers, et cetera. So let's. Talk about the basics of what you want to accomplish when we get when I think of all in a traditional enterprise environment, I think about secrets management.

Yes. But where it also goes beyond that is most people often think of it as static secrets. So I have a piece of sensitive data. I want to put it somewhere secure and I put it in a vault. One of the other big benefits of vault is what's known as dynamic credentials. So the idea of vault being able to act as a broker on my behalf to actually generate a set of credentials, whether it be to a database, whether it be to a cloud.

In this particular case, in AWS, I have credentials that I want out of AWS. So for those that are where the access key and the secret key for a given resource or given user or role that I want to use access. And so in essence, what's being done is vault can act on my behalf to request those given resources, because what I've done is I've delegated vaults as the trusted entity to request those on my behalf. So one thing, obviously, it makes traditionally portable.

So if I'm writing a application, a cloud based app, I don't have to use I'm not stuck with the AWS API or the Google Cloud API or Oracle API or whatever, or even in this case, the VMware API. If I need to create a ephemeral object and that ephemeral object needs credentials, I can call vault to do that. Is that the gist? Well, what it does, it doesn't have the abstraction layer. And so I would still code to the particular cloud.

But what I've now done is I've provided a mechanism for me to request short lived credentials. So think of a scenario where I need to do a particular operation in AWS. More often than not, what I would typically do is I might go on to an EC2 instance in AWS. Or what often happens is as an engineer, unfortunately, I might request go into AWS, generate a key pair, access key and secret key. And then I'll use that on my laptop to interact with the cloud.

The challenge becomes, well, what do I do with those credentials after 12 hours or three days or after 12 days? When you go into AWS and you see, hey, these credentials have been active for 2000 days, that becomes a problem from a security standpoint. And so I want a way to be able to access those same credentials and the capabilities. But in an ephemeral nature to where if something were if those credentials were to get stolen after six hours, those credentials are no longer valid because they're being automatically rotated.

So that's a really big benefit of the dynamic credential capability in HashiCorp Vault. So that's dynamic credentialing, which is really cool. You're reducing you. One, you're reducing your attack surface area. Someone can get credentials, but in theory, limited availability. So you're reducing your attack surface. And then the other thing I would imagine since Vault gives me the capability to create credentials and manage credentials. I now have identity within Vault. Yeah, absolutely. So there's a number of ways to authenticate with Vault.

So that's where it becomes that broker. And so one of the challenges that I faced was a number of platforms can integrate with HashiCorp Vault utilizing a trusted identity. And so what I mean by that is, let's say I have an instance on AWS. That EC2 instance has a piece of identity metadata that HashiCorp Vault can trust. So I set up the AWS authentication method within HashiCorp Vault. And that's saying that HashiCorp Vault should trust AWS when it tells me that there's an EC2 instance submitting a request.

And so when the EC2 instance requests to authenticate to Vault, it presents the information that's accessible via the instance metadata and says to Vault, here's my identity. Vault reaches out to AWS and says, is this actually an EC2 instance in AWS that has the appropriate constraints? And Vault says, yes. OK. AWS says, yes, this is an EC2 instance. Vault then essentially transitively trust the EC2 instance. The problem I had was there is no instance metadata for vSphere. Right.

And you're leading right up and I'm not going to interrupt you. Like what? Like when I think about vSphere and when I looked at what I can do with EC2, I can't do that natively in vSphere. Like I can't. I can't or I won't build against the vSphere API to do a lot of what I can do with AWS. It seems heavy to try and do that. What made you kind of go down this Vault route and what have you, you know, what's the outcome?

Yeah, for me, the challenge was I want I wanted a mechanism very similar to that where Vault calls them trusted platforms. So you've got AWS, Azure, GCP, even Kubernetes. All of these these cloud platforms, a cloud native platforms similar that have a mechanism built in where Vault can do that trusting. And so for me, it was I wanted I want something very similar to that for my vSphere virtual machines, whether it be able to access an AWS resource, whether it was to access even a database in my my local data center to where I would leverage.

In the high secret Vault nomenclature, what's known as app role. So app role is a mechanism typically for machine identity within high secret Vault. And so boiling it down, really, it's just a set of credentials that are passed to the virtual machine at provisioning. And so typically what we leveraged is an orchestrator like a puppet, a shaft, whatever automation tool, like an Ansible that you're using as part of the stand up of that workload. The challenge becomes is that the experience isn't the same as much as it gets close and some of that capability and functionality.

It's not the same in terms of that experience of authentication to Vault, as well as the ultimate experience of as a consumer. I want to be able to to utilize, in my example, native AWS CLI tools out of the box. That's the experience you get with AWS EC2. When I install something like the AWS CLI or I'm using utilizing an SDK as a developer, as long as I have the role associated with that EC2 instance and I've given it the appropriate credentials, I'm just going to run the command or run my code.

And it just works. It just works because there's the instance metadata associated with that EC2 instance. And so within the EC2 instance, I can hit an API endpoint that's only exposed to that EC2 instance to fetch credentials for that particular role and get the capabilities that I want, whether it's access an S3 object or pump data into an RDS database. That's the functionality that really transforms what people are doing with cloud to make it extremely simple from an IAM standpoint. And so I wanted to bring the same experience to a VMware vSphere environment.

We start to look at more and more on-prem data center environments are starting to bring in these cloud native type services, whether you look at something like an S3 object compatible storage capability where I want to do object storage on-prem. But now, how do I handle that identity to that object storage, whether it's an IO or a cloudian or a number of the other players in a fashion similar to, I want my vSphere VM to access that object storage, but I want to do it like like an AWS or Azure or GCP.

So I think one of the key. Concepts to get down here is that vault is identity proxy when we're using it in the ideal of saying, you know what, I'm going to bring two diverse worlds together. EC2 in VMware vSphere VMs, or we can go one step further. This kind of solves a problem that I've that I've been noodled for a couple of years, actually. It's OK. I have a Lambda instance and that Lambda instance needs to access data that's in Oracle database on premises.

There's been a couple of different ways to solve this now that I first thought of the challenge. But I could do the same thing basically with vault again. I could vault could front in the Oracle database identity so I can write my application to say, hey, use these vault credentials to access the VM. So I have my access control at a security level that's not necessarily integrated into developers and or network admins, et cetera, needing to control security at these legacy abstracts that we had today.

We now have this AWS like abstract vault to centralize security management. Yes, that we can't write natively to the Oracle API to vault, but we can leverage vault as the mechanism to get the credentials we need to do the authenticated work we need to do. Yeah, absolutely. And that was that was one of the reasons I looked to leverage vaults was because of its prevalence in environments, as well as the additional capabilities. And so the solution I built, which I sort of code name V off, looks to to leverage those benefits of a high secret vault and take advantage of that in a deeper technical level.

Really, what it's doing is utilizing VMware's guest info. So typically, those that are familiar with VMware, no guest info when they want to deploy an OVA for an OVA. How do I configure that appliance or that system that OVA OVF? Well, I add information that ultimately gets passed to the guest info, which is available within the guest operating system. And so the guest operating system pulls that information. And so effectively, what I've done is I interact with high secret vault utilizing the V off platform.

So when a virtual machine gets spun up or in this case, the piece of metadata custom attribute that I'm using for specifying the role that gets changed or updated, the V off platform is reaching out the high script vault, generating those credentials to authenticate, which are the app role credentials, and then injecting that into the virtual machines guest info. So now that the credentials to access vault or to authenticate to vault are in the guest info, now the operating system can access those credentials.

So that's where I initially started. Then I wanted to take it a step further to add that AWS like or cloud like experience. And so I wrote a quick agent or assistant, in essence, just a binary for the operating system that effectively serves as an instance metadata server. And so what it's doing is it's listening on the local system and it's set to the 169254169254 that AWS instance metadata is. And so when I run the AWS CLI, it's going to see that that actual IP address is listening.

There's a web server, a web service, and then it's hitting the web service. It's proxying effectively. It's reading the guest info, the credentials to access vault. It reaches out to vault, says, give me credentials for this case, AWS associated with the role I've specified and then responds back in the appropriate fashion to the actual API call. Thus providing that AWS like native AWS like experience. So this is super interesting. Where are you at? And this is like an open source project that you put out.

Is the source code available? Yeah, source code's available. It's something I've been tinkering on, actually, for the better part of probably about three or so years. Oh, this isn't just something you thought of last night. No. So it's something I've noodled on, on and off. Because, of course, the problem has always been has always been there from a VMware vSphere perspective. And at one point, VMware had what was it, VIO, the vSphere integrated OpenStack that tried to provide a an instance metadata server.

And it was doing it over some networking constructs with NSX. And, of course, those that are familiar with NSX know how complex that can get very quickly to provide some some mechanism like that. So I believe there's questions around scale, etc. Once, you know, scale breaks everything. So are you looking for people to contribute or is this mainly just going to be a one person project? I'm definitely looking for people to contribute as as they see some interest. I've definitely gotten some a couple of people here and there that have pinged me about interest in it.

One of the things that I've recently done was integrated some some CI testing with the actually the vSphere simulator, the vCenter simulator. And so I can start to test at some similar to scale as it relates to number of data centers, number of clusters, number of virtual machines in a simulated fashion, very close to what a production environment would look like. So I was always trying to ramp it up to a couple of hundred virtual machines just to kick the tires and see where it falls over.

So where can people find the project? Yeah. Yeah. So it's on my get up. So get up dot com slash Martez R. R. H. Of course, Vauth in alignment with a number of the vSphere naming constructs. All right, Martez. Thanks a lot for joining and sharing about your project. If you want to learn more about the CTO advisor, you can follow us on the web, the CTO advisor dot com. If you want to learn more about Martez, Martez, what's the Twitter handle?

Twitter handle is Green Read Tech, G-R-E-E-N-R-E-E-D-T-E-C-H. And speaking of Twitter, if you have questions for me, DMs are open at CTO Advisor on Twitter. Talk to you next. CTO Advisor podcast.