Kelsey Hightower VMWare Explore 2022

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

All right, we're back in business. I'm just coming from Moscone West. If you're not familiar with the VMware Explore Campus, three buildings, Moscone South, Moscone Center, and Moscone West. So it's not quite as bad as like a AWS re-invent where you're two miles from your next session, but I'm coming from a session where we talked to the Adbeats CTO and the CTO, a CTO from CloudFlare, really great conversation. You guys should go back and look at it. But we're on the verge of another great conversation.

Kelsey, you did a fabulous job with the keynote this morning, walking us old heads. I'm gonna call myself an old head VMware administrator. At one point I was a VMware administrator. I worried about the control plane. It's my job to administer the control plane, and I look at something like a Kubernetes or even a cloud control plane, and I'm thinking I'm a control plane expert for VMware. I'm not a control plane expert for anything cloud. I'm worried about moving.

And you just, you made me feel at home. Where did that concept come from, to just walk from old school virtualization to Kubernetes, walk us through that? I think for most people, if you start to respect the fundamentals, you have software. Software has an installation guide. It has a configuration guide. So we've gone from installing VMware email servers, database servers, application servers, and to me, fundamentally, it's all the same. You need memory, some storage, some CPU, and there's a config.

Now, it may go on a different path. It may need a different user to run as, but fundamentally, most software goes through the exact same process. And so for me, I stopped saying, hey, things like I'm a VMware administrator. I just know how to administrate this particular product. If you gave me OpenStack, there's some similarities. There's a control plane. There's things that need to be installed, and it's going to produce some VMs just like this other platform. So I kind of got into the habit of being a fundamentalist.

What are the fundamentals of my job? And I think when you get comfortable with fundamentals, it gives you a lot more courage to say, hey, here's a new software package, all right? Where's the install instructions? I expect it not to work the first time. I expect the configuration that I use on day one is not going to be the configuration I'm going to use three years from now when we start using more advanced features. So the fear of something new just went away.

So you said something pretty fundamental in our pre-discussion, and it just clicked with me. I've done a lot of data center migrations in my day. So when I look at migrating to the cloud, there's fundamental things I know to look for. We'll get into that later. So I fast forward to the specifics of the new underlay, the new control plane and those differences. And I kind of know that you do a really great job of not fast forwarding to that part, but thinking through the fundamentals.

So we're going to set up the problem statement. I've never moved a data center in my life. I've never gone from one virtual location to a new virtual location. My environment has been static the past seven years. And someone tells me that I need to migrate something from vSphere to Google Cloud, AWS, or somewhere else. How should I begin to break down that problem? So if you think about it, let's say you got a job and you've been there for seven years.

Most people have never built out a brand new data center. So this person has come and data centers are rebuilt. Power has already been provisioned. Network has already been put in place. VMware is already installed. They're just coming in to administrate. So they don't know how it got built up. They've just been managing it in place. And so when you say, hey, we're going to go to cloud. Think about it. You've never in your entire seven year career bootstrapped something from the ground up.

You don't know how to set up the perimeter, right? You don't know how to set up the power. You don't know how to set up the project because someone else has already done that. So when you go to Google Cloud or any other platform, you're like, what needs to happen? Do I need to create some servers first and install the control plane? Is this the control plane? Because I only know the tool that's in front of me. You've never broken down the fundamentals in your life.

And so I think that the step needs to be is like, wait a minute, we're about to bootstrap. We're not moving. We got to bootstrap first. So what's part of the bootstrapping process? I don't know where the storage is because NetApp was already here when I joined the company. I didn't make that decision. I didn't evaluate NetApp versus the competition. This was all choices made for me. And then you're going to ask this person, now look, this person may have been promoted five or six times over that period of time.

And now they're the director of data center and you've given them this new ownership task of carving out the cloud. They've never bootstrapped an environment before. So that person's going to have to step back and say, this ain't a migration. I got to bootstrap the cloud. The hypervisor's different. The console is different. My muscle memory ain't going to work for me over there. I'm going to be lost. And I think what you got to do is say, look, I'm lost.

I don't know how to bootstrap it. Anyone who's ever bootstrapped before, I need that education. Typically what you learn is once the bootstrapping's in place you say, oh, so now that that's in place, storage is just taken care of. Yeah, you ain't got to hook up NetApp and then add a plugin. It's just there. Okay, so how do we get a VM? And once someone shows that person one time, here's how you get a VM, they're like, oh, so this is the same type of VM that I got over here.

Ah, then they start mapping concepts and then they start asking a second wave of questions. Here's how I do firewalls in my environment. Can you show me how I do firewall rules in this environment? And that's when that person starts to catch on. Okay, it's different, but the fundamentals are the same because they finally have two different environments. So I think this is one of the challenges. We've talked about the difference in keynotes in environments. A show like VMware, they made a huge announcement for this community.

0. Everyone literally got excited because that's the thing that I need to design for and move to. I'm on this track. 0. I know what my job is for the next year. I know the community I need to go to and get answers. When you go to a open source conference like KubeCon, CloudNativeCon, it is, oh, here's this project called NAT. In my vernacular, NAT is like a IP addressing thing and why are they going into such technical detail about NAT, something that I have no idea what it does or why you might use it.

There is no set roadmap. So talk to me about kind of that community transfer, like going from this type of community where the vendor kind of tells you what to do next to a community that, you know, we have a box of Legos behind us. It's just a box of Legos. Go and build a Millennium Falcon. Wait, it doesn't come in a kit? What do you mean? Yeah, I think when you go to a vendor, you're looking for solutions.

When I go to a restaurant, I want a menu with a price next to it and you're basically giving me a limited set of choices of what we finna have, right? Because I went to the Italian restaurant for a reason and I'm not surprised. Now, when I go to the grocery store, I just want ingredients. I may not actually know what I want to eat for the whole week. So I want to see all the ingredients available to me. I might want to try something new.

I might want to explore. Ooh, Ethiopian. Okay, I've never made that before. Oh, I turned the box around. There's a recipe. Hey, we gonna try this. Let's buy all the ingredients and we gonna try to make Ethiopian at the house tonight. So I think when you go to an open source conference like a KubeCon, it's the art of the possible. I want to taste a new spice I've never tasted before. I want it to be a little off the wall.

I want to try something like, I don't know, you can put those two things together and you go there for inspiration. And if you're a vendor like VMware, you also go for inspiration because even though you have a solution, it doesn't do everything. There are gaps in all of these solutions and we know how this works. When we start gluing together a system administrator, one of our jobs is to take the vendor package and say, ah, this give me about 90% of what we need.

I had to go get a shell script and fill in the gap myself with a custom solution. Where we're at now in 2022, you go to KubeCon, that's what your peers that are making all these custom solutions that sit on top, right? Kubernetes had just came out. VMware was like, VM's everything. But then you're like, yo, what am I going to do for apps? What am I going to do for container management? Of course I can just install them on VMs, but there's no orchestration layer.

So at that checkpoint, VMware didn't have anything for you. You got to go to KubeCon to see what the next milestone is. But then I think the full circle comes where, all right, now that we understand the technology, now that it's been in production for a little while, VMware can say, you know what? We're going to add the VMware experience on top. So we're going to let you take your existing skillset and consume Kubernetes without becoming the expert. You ain't got to go to the grocery store and be a chef.

You can come to the restaurant. We just got a new entry on the menu. And I think people miss this part. 1 introduced the concept of namespaces. We don't have to go into technically the what is a namespace and the importance of it. But it is a commitment to making Kubernetes easier to be introduced into your environment. It is the way namespaces, like if you have seven years experience and most of it is in Kubernetes and Linux, you're like, what do you mean vSphere doesn't have namespaces?

Namespaces is just a basic computer science concept that should always be there. But this is the value that vendors like VMware, OpenShift, et cetera, and Google bring. They make these things more consumable and digestible. It's not just Legos anymore. Yeah, and I think people got to realize is that you can't expect a vendor to figure out everything that will happen in the computing landscape. There's room for disruption. And so one thing that any smart vendor does, I always tell people, the only thing worse than lock-in is lock-out, right?

You get a vendor, they do right by you, and look, you're happy to cut that check for that license. The thing that makes you upset, though, is you see everybody else doing containers, and you go to your vendors, like, oh yeah, we don't support that. You're like, well, that's what we want to do. So you mean to tell me I got to switch my whole entire vendor out? That'll take me years. And that's lock-out. What you want to do is see what you just described.

Your vendor goes to the KubeCons and they say, yo, namespaces, that's an interesting concept. That gives us another dimension of compartmentalizing a normal VMware install, and that might be the missing thing that we needed for better isolation and better self-service. And so when that namespace concept appears, sometimes that's a form of innovation. Taking something new in a different area and applying it to an existing area, that's what brings novelty to it. And I think for a lot of people that kind of stuck with VMware, they've been excited to see the right things at the right time come into their environment, and then you ain't got to learn the whole thing from scratch.

You just get to bring in the right primitives that make sense to you. So let's talk about kind of that day two. I've gotten that I need to learn the basics. I've got a community that I can learn that from. There's a, for those of you who don't know, there's actually a Kubernetes Slack channel. Like, you just got to find the spaces, and the VMware community is the VMUG user groups. There's this familiarity, there's these places to go. There's plenty of places for that in the cloud native community.

Let's continue to break down kind of the basics. I understand the difference between the control planes. I understand the things I need to set up. How should I think about triage? There are things that I need to do that are big. I've learned that if I take a monolithic application and just move it to a cloud native or cloud environment, it'll work, but I'm not going to get an optimal experience. I'll get some advantages, but I'll lose some advantages from having it in my VM environment.

I need to break this app up somehow. What do I get that next level of fundamentals? Or what should I look at when I'm looking at to decompose an app without going to bother a developer? There is no substitute for experience. Time, time. You can read about somebody else doing it. You can watch a movie of somebody else doing it, but nothing substitutes you doing it. And so even if you get it right the first time, honestly, that's the thing that robs people of the experience.

If I follow the directions to a T and we somehow turn a monolith into microservice and it just works without a hitch, well, unfortunately, you don't have any experience on what it means when it doesn't work. What you want in dev, it's like, yo, this piece is not talking to this other piece. Ah, you got to do this firewall rule. Well, someone got a proxy in between because we try and service mesh out. That is the best teacher, full stop. And I know a lot of people get started like, oh, I want the best practice day one.

I was like, listen to me. I can give you the practice, don't mean it's the best. It's the best for this scenario. And if you just copy and paste your way through, day 297, right before Black Friday, you ain't going to be able to troubleshoot anything because you have no experience of what happens when it goes wrong. And unfortunately, I think there's going to have to be a bit of a patience thing that says, we're going to have to exercise this thing.

And that's why I think it's smart for people to have that POC and break it. If you can't break your POC, you are not ready to go to production. That's what I mean by some level of experience. People are going from POC to production. I'm like, dude, that's too fast. POC is also for you to learn. You're supposed to send a bunch of traffic until it falls over. It's like, oh, this ain't ready for prime time. And then there's a feedback loop that goes with it.

So my advice to most people would be, if you're going to the cloud for the first time, your biggest challenge is a new environment that you're not used to. Don't break up your app at the same time. You can't fight two battles at the same time because breaking up an app is hard even in an environment you're familiar with. You'd be like, yo, I don't know why the performance has dropped. Yeah, we did, for disclaimer, we did a sponsored research with Google in which we took a monolithic application, broke it up, migrated Google Cloud using Google tools.

This isn't about that, but it's about that point you just made. We actually didn't break it up in Google. We broke it up before we moved it. The assessment tool said, oh, you know, there's a NFS share that you should probably take care of before you move it. There's a web server running on that. You could probably break that web server out, get it to work in an environment that you know, that you completely understand, and then move it. Or move it to an environment that you're going to get to know, as is, in a way that you can support it and understand it.

If you need to move it as a big VM, move it as a big VM, learn a new environment. I'm a big fan of lift and learn. I know people say lift and shift is something you should never do, but nothing's wrong with lift and learn, because even if you took the same app, unmodified, and you tried to put it on a VM, there's going to be things that are different, like the metadata service, the way logging may or may not work, and that's where you want to spend your time enhancing the app and focus on getting that done, and then you can move to the next chapter.

I tell people all the time, some of the nuance that they don't see when they say a VM is a VM, VMware made this big kerfuffle about this new management solution, which is partially vRealize, which gives you a slew of information about IOPS, disk performance, server health, et cetera. You move your VM to the cloud, you don't get any of that. You might get IOPS, I don't know, but you don't get any of that underlay information, and that's stuff that you intuitively knew in your private data center.

Oh, transactions have slowed down. Oh, that's because disk has gotten full. Or it's because you're on a virtual CPU with a shared core, because you wanted to save a bunch of money, and your process is being frozen, because that's how computers work when you're in a shared core environment, versus the full i9 that you got to one application. Remember, it's just these things that people have taken for granted over time. If you're the only one on a 64-core box with 120 gigs of RAM, and you move into four gigs of RAM, one virtual CPU, you should expect some differences.

So this, again, goes back to, let's bring it back to this core, which is the fundamentals, bringing it down to the fundamentals. I don't think we, I talk to, I advise the company without getting into a full engagement with them. They came to me and they said, you know what, Keith, we moved to a distributed application development model, and the app is breaking more than it was before. And I asked them what was their previous process. We would, it was a monolith.

We would submit all our changes at one time, we'd test on Thursday, and then we'd deploy the app, but we need to be more agile. Well, what's the new process? Well, everyone kind of does their thing, the text in a vacuum, and then deploy. I'm like, well, you don't have a process. You don't have a testing process. You had one before, you didn't intuitively, you didn't write it down and you knew what you did. You didn't document it. Now you're running into an environment where you're not testing.

Well, so this is what I mean by fundamentals. First of all, you gotta understand the big picture. Most apps are written for a very simple set of purposes. Like the fundamentals of computing are, you have logic and data. You have an e-commerce site. There's a form to generate data, like an order, and it goes to the app to process the order, and ideally you're gonna put the results in some database and return a response. This is the high-level, big-picture fundamental.

Now, you might say, well, how fast does the website need to be? Hey, after three milliseconds, our customers leave and never come back. Three milliseconds, that's very fast. That's like memory access fast. If you break the monolith up into three pieces, the order services over here, the catalog services over here, we can just do the math on the envelope. Hit this service. Let's say you get lucky two milliseconds, just close to the load balancer. But to hit the other service, that's gonna cost you another three to five.

That's gonna cost you three to five, and you're gonna blow your budget. I think a lot of times, people can figure these things out on paper. These are fundamentals. If we break the app up, there will be more latency. If we put security between the new components, there will be even more latency and higher costs. Microservices are not a best practice in general. It is a trade-off for another bottleneck, and it's usually a large group of people trying to work on the same solution.

You know what? To give them better ability to write modular code in their own, maybe, repository, the big trade-off is, then deployments are about to get real complex. Security's about to get real complex, and that's not the right trade-off for everybody. So, the last point I want to make, we've been talking at very practical levels from get the basics, and I think what a lot of people understand intuitively, is I don't even know how to get to the basics. You guys are asking some great questions, but I don't know how to even ask those questions.

And just like there's no substitution for time when it comes to getting, testing something in a new environment, there's no substitution for time to having these conversations with mentors and peers who can challenge you and direct you in a way. You need to take what you think, excuse me, you need to take what you think is your process, take it to somebody who sees it, and have them say, hey, what did I miss? Yeah, I think zero trust is a good example. When you hear zero trust, you're like, wow, what does that mean?

And so, for me, I'm the type of person that wants to break it down on a piece of paper. Let's say I wanted a zero trust for a JIRA install. JIRA is my issue tracker. It connects to a database. I'm going to start there. And so, here's my current deployment. Anyone can log into JIRA. JIRA has a username and password on a disk. It loads that password up, and it connects to the database. Where's this setup falling short of zero trust?

And then when someone draws or helps you draw the, well, here's the components you would have to add in all these layers to remove implicit trust, he's like, oh, so what I currently have, here's the delta, here's the dip. And I think that thing in the middle, that's the understanding part. You want to start with some foundation, and then having someone walk you through, and then honestly, I'm the type of person that's going to take it to two or three other people and be like, hey, is this zero trust?

They're like, nah, you're missing a thing. That's close, but you need to do this instead. It's like, ah, I didn't think about having a laptop have a certificate so when it connects to JIRA, I'm authenticating the user and not just the username and password because that can be shared. We call this just peer review. This is the engineering process. But that's what's missing in our field. A lot of people ain't doing engineering. A lot of people are cargo culting.

And there ain't nothing wrong with copy and pasting, but this is your industry. This is your career. And that means you've got to be able to break things down. It's not a week-long process. This is like a multi-year process that every opportunity you get, you say, hey, let me break out my research. I'm at this level of understanding. Am I missing something? And you'll get there where you say, oh, you're not, you got it. And I'll share the quickest experience from my career.

I started at Lockheed Martin back in 2012. And they said, Keith, we're gonna put you in charge of the lab. I'm like, oh, great. And I'm like, well, what's the biggest problem? Ah, we're having a hard time getting it off. I'm like, well, what's the requirements? They ran it down. Oh, I can do that in two weeks. You can do that in two weeks? What do you mean? So I went to go do the thing, and they were like, where's your requirements, Dr.

McKeith? Where's your, who peer-reviewed it? Where's the defect tracker? Where's all these engineering artifacts? And I had to, for the first time in my career, I had been in IT for 15, almost 15 years at that point. And for the first time in my career, I realized I was not an engineer yet. And I swallowed it, and I spent the next two years learning proper engineering processes, and it's been tremendous to my career. I can now look at things like Kubernetes, OpenStack, previous to that, and VMware, and say, you know what, this is just a control plane.

It does, it has a job, it has a thing to do. There's some technical stuff that I'm missing to understand it completely, but it is what it is. And it's okay that new things take time to learn. I think a lot of people get frustrated if they can't digest everything in two or three days. I'm like, no, no, no, that's not how this works. So you got to make sure that you're investing in something that's worth learning. So Kelsey, I appreciate you taking out the time.

People ask me all the time, this is maybe the fourth or fifth thing that Kelsey has done with me. Keith, can you introduce me to Kelsey? I don't think you really understand what his role is in the community. And we'll leave that until, like, how should people engage the great Kelsey Hightower? Well, honestly, I don't see myself as a great Kelsey Hightower, but I do respect people that support the stuff that I'm doing. People will come and introduce themselves to me.

And I will say this, I'm trying to learn from you too. So don't be surprised if I ask you about what's in your world. I don't care if you are just starting out because you have an advantage of seeing the world from a lens I can't see. And so I would say, look, when you engage, I appreciate the support. I'm going to always say thank you, but give me an opportunity to learn from you too. And I would say that's the best way to engage.

And if you want to engage with me, just simply hit me up on my DMs. I'm not as busy as Kelsey and I can answer questions. I can even tell you how to approach, how to exchange value with someone like a Kelsey Hightower. So you're both getting out of it. We're here to support the community. That's what the CTO Advisor is about. You can follow us on the web, the CTO Advisor. Make sure to stay engaged on our YouTube channels.

We continue to talk to developers, maintainers, operators, analysts, and help you do your job better. Stay tuned for more coverage.