CTO Advisor 063: Can enterprise IT adapt? w/Kelsey Hightower

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

Hey, you listened to episode 63 of the CTO advisor podcast with our special guest, Kelsey Hightower from Google. If you follow the Kubernetes movement at all, you know that Kelsey has been a great contributor to that community. We'll get into some great, both technical and high level details around Kubernetes. Special thanks to my co-host Mark May for roping Kelsey into doing our little podcast. Before we get started, I want to thank our podcast sponsor Druva Inc. According to Druva, Druva delivers the industry's first data management as a service, a single SaaS platform that unifies data protection and management for endpoints, infrastructure and cloud applications.

Unlike traditional systems, Druva aggregates this business critical data for scalable backup and disaster recovery, while also unlocking the true value of search and advanced analytics for governance of that data. For administrators, Druva means simplification of data management across the entire enterprise. For IT leaders, Druva means scalability without limits. For the CFO, Druva means saving a boatload of money through intelligent storage. For InfoSec, Druva means security by design to the highest standards. For VMware administrators and Nutanix environments, Druva means unified backup, disaster recovery and archival, powered by AWS.

Druva roughly translates to saving your butt from data loss, litigation mishaps and regulatory fines. Druva, the future of cloud data protection. Now our discussion with Kelsey. So Kelsey, for those two or three people who don't know who you are and who you work for, why don't you give a brief bio? Hey, so I'm a system admin turned developer. Work at Google now. Distributed systems is my jam, mainly around the Google Cloud platform, but particularly interested in all technologies from programming to system administration and how we can actually make this stuff a little bit better.

So Kelsey, many of you have met in person once at a Intel event, and I was really struck by like your thirst for just crazy amounts of knowledge. Like you wanted to know all of the telemetry data coming off of the system and processor and then be able to do something with. So when you say that you have a wide set of interests, that is absolutely the case. Then we got into a detailed podcast with Greg Farrell at the Pocket Pushers about networking and containers and why networking inside of containers is hard or shouldn't be hard.

So we can almost go anywhere in this conversation, anywhere specific you want to go. Well I think a lot of people, if you're a CTO out there listening, you probably are very interested in this whole container movement. And it's bigger than just, let's say, containers, the low-level technology, but this different way of working. You know, five, six years ago, people were talking about the cloud and this opportunity to leverage a cloud provider's infrastructure to go a little bit faster than what you can do on-prem.

And now that conversation has moved towards the app level, right? We're not talking about machines and networking. I mean, some people still are. But the new opportunities that present themselves are around the stuff around containers. We have serverless. And what we're doing now is getting closer to the actual value part, right? You wrote this in application. How do you get this in front of your customers? And now do it at the cheapest price probably ever possible. So Mark, you're a CTO-level resource.

You manage a fairly large team in your day job. Is that a fair assessment? Is that kind of where your questions lie when it comes to cloud and the next phase of applications? Yeah. I think we're all trying to find that new value line as we realize more and more that infrastructure is just an enabler, right? That's never the value. As we move towards that build versus buy and that we've gone through that transformation in the infrastructure, but now we're trying to do the same thing in the application space.

And we're trying to figure out how we do that in a new way. And I think containers are playing a huge role. But I think some things that a lot of people, especially our audience, misses is container orchestration is extremely important. And you don't hear so much about that at the C-level. So Kelsey, I guess the question I would have for you is walk us through the relationship between containers, applications, and basically the cloud, that very high-level approach. What does containers enable at that low level?

How does that help application development and support? And then how does that translate to the ability to leverage the cloud? Yeah. So if I think back to what you got in your data center, right? So if you close your eyes and you look into your data center, you hear all the machines humming, hear the fans going, that's your power bill spiking right there. And we know what to do with machines. You put an operating system on it, you get your configuration management tool of choice, and you configure and you manage that server.

And a lot of the lifecycle of most IT people revolves around the server, the machine, who can access the machine, what runs on the machine. And when the cloud first came out, people just took that exact mindset and just did it on someone else's cloud, right? The meme was the cloud is just someone else's computers. And for the most part, the way people treated it, that was true for most people, right? It was just another machine somewhere else, and you brought all of your tools, all of your practices, good or bad, to another location.

And for most people, it didn't get drastically better. You don't have to order a physical machine to get shipped to your data center, but everything else kind of remained the same. So that was thinking very low on the machine level, and no one enjoys that particular job. So not to interrupt you, but Mark, we had this conversation with the folks over at Pillarwood, James Waters, and his customer, what was it, Express Scripts. And they were having the, they had the exact same conversation that, you know, this, we were stuck in this machine-based mentality.

And even with the, to Kelsey's point, the onset of cloud didn't fix that. We still, you know, took that same construct and brought it to the cloud. So I'm hearing some consistent messaging, at least from that application perspective, that, you know what, this machine approach is kind of maybe the broken way to approach it. Well, we've gotten better over time, right? You know, we've learned a lot since the last 10, 20 years of doing this. And I think if you talk to the Cloud Foundry folks, it's like everyone just wants like a platform, you know, a set of opinions that say, you know, this is how I put my application in front of customers.

This is what we're getting at. And I think the middle step to that is better infrastructure underneath. So this is where that container comes in with less opinions than a Cloud Foundry or App Engine or Heroku and says, how about this? We package your app with all of its dependencies, kind of like we do on a mobile device. And then we deploy it to any other platform that we want. And that's the difference in the conversation. You're not deploying to machines per se, Red Hat, Ubuntu.

Now you're deploying to platforms, Docker, Kubernetes, Mesos, or something else. How do we do that in the enterprise when everything we've done for so long from CMDB to the purchasing cycle is all revolving around that server? That's a big change for an enterprise to undertake, I would imagine. Man, that word enterprise, I think we should do a whole show of what the word enterprise means because to me, these are the same people I grew up with, I work with. There's nothing special necessarily about the enterprise other than when I hear that word, the only thing that triggers in my mind is that you have so many people and the more people you have, it's really hard to introduce a new way of doing things.

Let alone technology, just a new way of doing things. If your job is to order the server, that's what you do, you order the server. If you tell me to switch from ordering the server to figuring out what's the best platform to use, so take away the machine, that's going to require some time to get up to speed and learn all the things that took you maybe a decade or two to learn on how to purchase physical equipment. I think what we're seeing is in the enterprise, number one, if you have such an existing investment in technology, I'm not going to call it legacy, an existing investment in technology, most of the stuff that you have today got that way because of some good decisions, a lot of bad decisions, get this done by Friday.

A lot of the stuff that runs currently in enterprise isn't the holy grail, it isn't what people are happy about, it's the stuff that survived all of those last minute decisions. So it's pretty fragile in some cases, some people got it well. So now you introduce this enterprise and you say, hey, over the last 10 years that you've been building what you have, there's a whole new set of patterns called cloud native, we don't do this anymore, we don't do that anymore. So now when you write your new application, you may say, wow, with this new game knowledge, I am not going to make the same decisions or I'm using a library that just makes the better decisions out of the box.

And I think that's where the friction comes in, right? You're rebasing on top of new ideas that have survived over the last couple of decades. So I think the pushback with that has been from a technical perspective, we know exactly what to do, making a transition from bare metal servers to virtualization, from virtualization to containers really isn't that tough of a transition. If you have the skill to make the transition from client server to any other technology, you have the technical skill to make the transition to either a serverless concept or a container focused concept.

I think the problem that, quote unquote, the enterprise runs into is just the scale of inertia. And Kelsey, we've talked about this in the past, just one, the fundamental gap in knowledge of having solid computer science discipline and that ability to apply that discipline to yet a new delivery mechanism for delivering the end product, which is an application. So I guess the question is, is today's, quote unquote, enterprise equipped to make that change? You are. It depends on how under threat you are, right?

If you're on the brink of no profits, share price is going down, you can't retain talent, you can't compete in the marketplace, it's a whole different conversation. No one can say in nine or 18 months, we'll get to that. So it depends on where you're at. If you're in a stable place, maybe you've got a good position in the market. You know what, the existing technology you have, even if it isn't better or faster, may just be fine. And you know what, you should just stay with that.

But some people don't get the luxury of doing that anymore, right? 4 in production, people probably own all of your data at this point. So sometimes security forces your hand. We see this a lot in enterprise, right? We're not updating anything. Now there's a security vulnerability. That's not even a decision you get to make. You will upgrade by Friday. So I think the sense of urgency for enterprises, it all depends. If power starts to go up, if it's no longer affordable to have a data center or you can't build new data centers where your business needs to be, this is where change is forced upon you.

And I think that's kind of some of the friction we see where people think that they had control. Maybe the last 10 years you had a bit of control to move at your own pace. Now your customer's like, where's your mobile app? I want to be able to have voice assistance to talk to your particular product. So you're no longer in the driver's seat. The customer's in demand. And if you think you can just hang back, your competitor isn't.

And I think this is the difference we're seeing now and this is why the enterprise feels a bit tense. So from a monetary standpoint, most enterprises have the money to be able to pay for the talent they need. I think the problem is that they may undervalue from top to bottom who can make an impact, right? A centralized group of people who can decide where the company goes next. I think in this new competitive landscape, the innovation may come from the person who just joined six months ago, hands on the keyboard.

So the mark, I call this the syndrome, you can't teach somebody how to be hungry. You know, you can't, if you try to describe hunger to someone who's never been hungry, they just have no point of reference. So it sounds like what Kelsey is saying, that just out of sheer natural just need, the enterprise has that capability because they want to survive. Yeah. When your backup is up against the wall, you tend to be able to do what's necessary. I think that makes sense.

Yeah. And we've seen this, right? I mean, we've seen enterprises just come out and wow you, right? Like, you know, think about the banks. Everyone says the financial institutions would be the last to adopt certain technology because they have to play it safe. But if you look around, all the banks have some of the most innovative technology. Their mobile apps let you send money from your phone, check your balances, real time credit scores, real time fraud detection. So I think now the banks are starting to enter a world where it's super competitive because now they're competing with startups.

This is new for the banks. You got Stripe and Square entering their domain and doing things at a rate of speed and innovation that wasn't possible or the banks didn't think were possible before. And then you turn around and here we go. The banks are matching that pace and it's sometimes outperforming. So I think it just depends. Like, if you're really serious about being competitive in a market or you have to be competitive, you usually rise to the occasion. So that begs the question.

If I'm in an industry where I'm not being, you know, threatened, it's not ultra competitive right now. But we see that potential for that competitive landscape in the future, what can we start to do today? You know, what can we start to have our IT staff do? What can I do to get to a place where we're doing IT better? So I do a lot of exec briefings now. So I actually have learned to kind of balance the low level details with the high level perspectives required.

If you're an executive, if you're at that C level role, one of your jobs is to be forward thinking, right? You need to have a little bit more vision and that is kind of part of your duty. And if you're going to have a little bit of vision, it's not about focusing on keeping what you currently have running, right? That's a manager. If you're going to be an executive, then you need to look forward thinking. For example, if you're working at an insurance company and you're an executive, well, one thing to do is just keep an eye on the spreadsheets.

Are profits looking good? Great. Are we not paying out too much in claims? Fine. But if you were forward thinking, you would say, man, I could actually service more customers if I adopted some of this new technology responsibly. Example, there's drones. There's drones and machine learning. That sounds like a good way to start to test out ideas around doing claims adjustments by sending drones out to a site, taking pictures of a vehicle and then processing it against the thousands and thousands of images you've accumulated over the years and then build a model that says, hey, as an insurance company, we know what a total car looks like and we can train our own model to give us an advantage on response times and customer service that new insurance companies may not be able to do.

And to me, if you're an executive, that's what forward thinking looks like. So I love that example of net business capability and the application that supports that. So back to kind of the original question of the different phases of getting there, what's the whole bag of that presents I'm sorry, that prevents enterprises from jumping to that capability tomorrow? OK, that just sounds like an app I'll have my developers build it now. All right, so I'll give you from just my experience.

So I've worked in enterprise at various levels and going back to my individual contributor days, I can remember coming in and being indoctrinated with the rules. This is how we do things. Learn it. Well, do it well. And that's it. And a lot of times you destroy one of the best advantages of bringing in new people, new perspectives. Right. What you want to do is try to create a little bit of space, maybe in the first three to six months where you say, hey, after being here for a little while, what do you think about our current processes?

Do you have any new ideas? All right. That's the value. Like bring me new ideas. I just don't want bodies in the chair. So the biggest factor that I've seen at an individual contributor level was this. Everyone is just a resource. Your headcount. I have 15 people in my DevOps department. You're all interchangeable. That mindset just really destroys innovation or the lack of creating a place to try and fail seriously. Right. Like at Google, we have these things called like big bets.

And you want something that sounds crazy, that most executives is like, I think that's a bit too risky. Well, you know what? You need a few of those. Do them responsibly. Give them a budget and let those people work on an island to try to impress you or do or beat the odds because your competitors definitely trying to beat the odds. So as an enterprise, you have the ability to do this from a capital space hiring perspective to go out and allow people to do what you maybe can't see.

So, Mark, your perspective, what from a large enterprise perspective, what needs to change in the enterprise from a practical perspective? Where should CTOs, senior architects, where should they start that journey of transformation? For me, it's always now about understanding the business that IT is bringing, understanding the value that IT is bringing to the organization and investing where that value actually is. So in most cases, I think that's in the CIDC pipeline. The application elasticity pipeline, not necessarily and definitely not in what you've been doing the same way over and over again for the last 20 years.

So I would jump in, let me jump in like CICD is a done deal, like 10 years ago, done deal. Like I know a lot of people are talking about DevOps and CICD. Trust me, that is straight up done deal. There is nothing to figure out in that domain. Now, there might be some culture changes. There might be the underinvestment at your organization on the number of people who can do it or who are allowed to do it because some teams have gotten a special pass that they can continue to build out their laptop and they keep getting that VP exception to do so, holding kind of this discipline back.

But to me, that is the table stakes. CICD, DevOps, we need to say, listen, figure it out. You got six months to figure it out or go work with another team outside of this company to figure it out. Well, there's so much more available. I think that's a problem. What's that? People aren't doing that in a lot of organizations. They're regulated to doing what they have always done. And the status is not quo, right? So you can't keep doing the stuff we've always done.

It's not going to work anymore. Yeah, so I think that's the and as a leader, I think that's one of the most difficult challenges is to help impress upon the sense of urgency of change. S. car industry, it took the whole industry to kind of go bankrupt before there was a change in motivators of that bottom staff. It didn't it took, you know, that bottom staff to get kind of laid off. So we're you know, we're we're venturing into more of people management and inspiration.

But I guess that's a practical question. How do we inspire people to change so that they support that eventual mission? Because there's a lot of inertia. You know, I've been a DNS administrator for the past 15 years and I've made a pretty good life at it. Why should I learn how to you know, why should I learn infrastructure's code? I've I've updated these DNS entries manually and it provided a pretty good it provided a pretty good living for my family.

That's a really hard mentality to change. I honestly think if you're still serving the business with that flow, that's fine. I don't think you have to change for the sake of changing. I think one thing that leadership can do, since this is kind of kind of focused towards that leadership audience, sometimes you just got to go and break the tie or just say, pick one. All right. You know what? If you guys are examining five things for six months, I'm going to say, you know, we're going to do we're just going to use Jenkins.

That's it. Don't look at anything else. I don't care what's actually better. Right. Like maybe you give people a little bit of time to kind of make their own decision. But the truth is, at this point, most technology is pretty even in terms of base functionality. And when you're starting out, that should be your core focus. Get it in there. Get it done. And then V2 can come later. And I think people spend too much time trying to boil the ocean.

So if you're a DNS administrator and your company has one DNS server, you know what? You might be fine. Now, if your company comes back and say, you know, that last DNS outage cost us 10 million dollars, then your responsibility as an IT professional is to say, you know what? I'm going to turn in my DNS card and we're just going to use dying DNS with global DNS servers and call it good. And I think that's one of those things that's in our wheelhouse.

A lot of these companies are trusting us to make that call. When do I need to pivot in terms of what my role is to the company and think about adding a little bit more value even if no one asked me to. So, Mark, this goes back to your bill versus buy perspective from a technology, we technologists, we know what to do. We this stuff isn't necessarily new to us and what needs to be done. There's a culture change, but there's a from a CTO perspective.

We need to understand where the value is at. Where should I dedicate people resources to versus saying, you know what, I'm going to go to either the cloud or to my vendor or to service providers to make that offering. What do you where do you see some potential challenges or even things that executives really need to take a look at and be careful with it when they're going down this journey? I think from my perspective, it's competing priorities, right, that's what it always comes down to doing what we what we need to do to keep the business up and running and innovating when there's a finite amount of time that we have.

So we have to identify what's going to provide the best business value and do that. All right. So I don't think there's any one answer for anybody. It's to do what is best for the business. All right, with less, we're getting to that point that we need to close out, I can most definitely go down this route and Kelsey, I think we're going to take you up on this offer to talk about the definition of enterprise where you're not the first person to push back on us or push back on me on, oh, you know what, this concept of enterprise or, you know, web scale company versus non web scale company or tech company versus non tech company.

But until we have that conversation, where can people find you and more about what the good work that Google is doing? Well, I like to hang out on Twitter. I call that my global chat room. DMS are wide open, happy to help people with any of the core, low level details of the Docker, Kubernetes, you name it, anything around cloud. That's my primary place that you can find me or sometimes at a conference speaking about this technology. Then, Mark, I know people can find you on the Twitters at Sissy Storage, any blog you blog you want to post or plug, you know, you can always find my blog at virtual storage zone dot com or listen to this podcast.

And there you go. And you can find me on Twitter at CTO Advisor on Twitter. And you can, of course, find the blog, the CTO Advisor dot com. Talk to you guys next episode.