Preparing the Enterprise of Kubernetes

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

Hey, how's it going? It's Keith Townsend from the CTO Advisor. You're listening to, let's call this episode 107 of the CTO Advisor podcast. You know, we're going to continue our conversation around Kubernetes. And on the other end of the line, I have Kubernetes expert, Damani Corbin. You might have seen him on some recent CTO Dose videos. Damani, welcome to this format. Keith, thanks for having me. I appreciate it. No problem. So, Damani hit me on his program at, I think it was at KubeCon, right?

Right, right. And we started what was a really great animated conversation. Damani is passionate about Kubernetes. And I'm famous for saying that I hate Kubernetes. So, I'm going to give some context here because I tweeted this the other day. I've invested more money, more of my own money, in the Kubernetes ecosystem than I've done any technology. I support, I've sponsored the OpenFAS project. I've amplified all the Kubernetes content that I've created. I've done something that I normally don't do.

I've actually put paid reach behind content that I've done around Kubernetes. I believe Kubernetes is a very, very valuable tool for the enterprise once it's mature enough. And my objective is to push it towards maturity. Where I see Kubernetes is solving stateless problems that we might have in the enterprise, not necessarily supplanting VMs. But, and more specifically, the way we manage workloads using VM tools such as STVMM from Microsoft or VMware vSphere from VMware or Red Hat's virtual machine management tools.

Damani, where do you see, like, Kubernetes' role in the enterprise ultimately? Like, what are the advantages? Why do you promote Kubernetes? Keith, if you look at it, I think from the inception of Kubernetes, what happened inside of Google was the industry wanted to replicate what Google was doing with Borg, right? And the early folks involved had to try to figure out, how do I replicate that? And I think that's where Kubernetes was born. So it was born to replicate what Google was doing with Borg.

And simultaneously, I think it was, it came to fruition to combat what was happening in the cloud wars to push back against virtualization or to be another answer to the cloud war because Amazon was running away and leaving everyone. So Kubernetes was a technology that is the only way to level the playing field in the cloud. But it makes logical sense in the fact that it's completely open and it is going to be the de facto application platform for organizations. So I want to push back on the idea that Kubernetes is going to be the de facto platform for organizations.

NET. There's a bunch of, still a bunch of provisioning of new virtual machines to either create applications or extend applications that they get, COTS products that they get from enterprise software companies like SAP, Oracle, et cetera. There's an awful lot of that type of development going on. There is some container, some container development going on, but not at the scale that I see the enterprise leveraging Kubernetes as a central piece of managing that workload. What's your experience? Because you have a different worldview than mine.

Yeah, so we have to take a step back and realize that Kubernetes will become so low that developers don't have to worry about it. And the idea is that all applications aren't suited for Kubernetes. However, everything is API driven and the construct of Kubernetes allows it so that majority of these COTS applications that you speak of will start delivering it to customers via Kubernetes. So what's going to happen naturally in the environment, the enterprise architects and the operation teams are going to now have to react to managing these applications in a new way that they haven't had to in the past.

So let's talk about the challenges that come along with, because I think I agree with you, the driver is going to be COTS applications being delivered as containers and being deployed using Kubernetes. Today, where Kubernetes stands today is that if I'm going to deploy an application, generically speaking, just because where we're at today, what I've seen is that is going to be a one-to-one mapping. I'm going to have a Kubernetes cluster mapped to an application instance or to an application deployment. I'm not going to today deploy my SAP provided COTS application in the same container that I deploy or the same management platform that I deploy my typical application in because Kubernetes just isn't there from a ecosystem perspective.

Do you agree with that? Yeah, I agree. So the problem that that creates for the enterprise is some practical stuff. Let's take a look. I discovered Linkerd during the latest KubeCon. Extremely powerful service mesh that also provides a lot of application debug tools is great. So when you deploy Linkerd in a Kubernetes cluster and it's part of your application, a bunch of tools. The problem becomes when I go to expand my reach of leveraging Linkerd across multiple instances of Kubernetes.

That doesn't scale operationally within an enterprise. What I mean by that is saying that, you know, Linkerd deconfiguration pointed to one Kubernetes cluster or package as part of one set of applications isn't going to have the same settings and config files as another one deployed by typical folks. I think that's my biggest challenge with Kubernetes, just the disjointed nature of the application. Where do you think or where do you think Kubernetes needs to go or for customers looking to deploy Kubernetes today at scale?

How should they be looking at Kubernetes from a enterprise architect's view? So I think we need to take a step back and understand that there's three different constituents in this equation. I think primarily you want to speak to the C levels and there needs to be a business value that needs to be explained as to cloud versus cloud native. And I do a bunch of chalk talks, whiteboards on that. They need to understand the difference between cloud, just live and shift and actually refactoring applications.

And that's a high level conversation for executives. I think secondarily, there are developers that need to understand the value that they bring to the table and you don't want to waste developers time. You want to make it as easy as possible for developers to push applications and developers want to see their code running in production. So how can they quickly increase that feedback loop so that they can continually push applications? And then the third is operations and enterprise architects. And I think that's the environment that a lot of folks get stuck in.

And I think that's where you're stuck, Keith, like you're coming from that perspective and that's messy. Kubernetes is complex, but for developers and for and for C level, it makes sense. It's for the operations team to go figure it out. And let's talk about how we present it to the business so that this platform is robust so that we can deliver value to customers quickly. So there has to be a breaking apart of the conversation. If we're trying to approach it as a whole, we're not hitting the right audience and we're not landing the right messaging for the right people.

So I really love that breakdown of the three different audiences, and the three audiences must work in tandem. So the developer simply can't say that I'm going to use the tool that's most convenient for me, and I'm going to drive the process because it breaks. You can say that, you know what, yes, I can develop applications the fastest, but I need to care about production and I need to care about skill. This is what I've seen in a typical enterprise. You know, you can insert whatever technology you want as far as the hype cycle.

It doesn't have to be containers. It can be cloud. I'll give you a great example of the challenge that I'll typically get involved with in the enterprise. A business group will go out and they'll build an application, and they'll use tools like AWS Lambda. They'll use things native to AWS, and then they built this great tool that will solve a business challenge. You go give that to operations to operate, and it looks nothing like what they've produced in the past or that they've supported in the past.

And you say, hey, manage this thing that I built that doesn't comply to any of the standards or any of the processes we've used in the past. 5 million or $2 million to hire the staff and create the processes needed to do that. And they say, no, we don't have that budget. And I say, well, I can't support your application. Go out and get support from the person who built it. That's typically how enterprise operations works. We get funded by being able to spread our costs across several applications.

When developers don't conform to those standards, you create costs that just don't work. Right, so I 100% agree with you. I think what needs to be understood is that there needs to be a culture change so that development teams don't do those type of things. So a decision had to be made, and a majority of organizations have already started on a Kubernetes path, a Kubernetes journey. Operations is responsible for setting up an infrastructure and some sort of a platform that enables the majority of applications to have an onboarding ramp to Kubernetes.

So that development teams, like the bespoke one you just spoke about, does not go ahead and do their own thing. They have to follow the guidelines to be able to run effectively and that there's consistency, compliance, audibility, all of these things that are necessary involved from the onset. So operations has to step out of their box and be forward-thinking because the developers are going to run away and leave them, and then we're back in a cycle. In my opinion – go ahead.

No, I love that. So I don't disagree with you on that. I agree with you 100% that operations – if you've decided that Kubernetes is going to be any significant portion of your operating model going forward, whether you're going to continue to develop and maintain virtual machine-based applications in addition to container-based applications, or if you're going to go all container, it really doesn't matter. You need to – operations needs to get in front of it. My challenge with Kubernetes when it comes to Kubernetes in an enterprise is that Kubernetes doesn't have the ecosystem around doing that at scale for the diversity of applications that's in the enterprises.

I think my pushback at it, like if I wanted to adopt Kubernetes today at that scale, is it reasonable to think that I could do it? Are you familiar with GitOps? No. So there's a methodology called GitOps where Git is utilized as a single source of truth. There's immutable firewall in between. So developers have been familiar with Git for ages now. Operations teams have been utilizing Git more and more often in the past couple of years. So developers check code into Git.

Operators go and make a copy of what was placed in Git for them to then run in production. This is a methodology that is catching on like wildfire, and every major provider has been putting out a GitOps strategy as a way to manage the complexity of Kubernetes. And it's not just Kubernetes. It is any type of declarative infrastructure. So because Kubernetes is declarative, you can describe to your operations team via YAML file everything that your environment needs to be or look like, the desired state of it.

If that's put into Git, the runbook gets eliminated, operation now comes, and your complete operation can run out of Git. So I've heard that I've not heard of it called GitOps before when I'm doing DevOps for infrastructure. Git is just one of the – is that declarative subsystem that I use to manage state of configuration. So if I have a set of PowerShell scripts that I use to manage my infrastructure, I use Git to do version control of that. So you're saying GitOps from a broader perspective, I'm using that same subsystem to manage the declarative state of the applications that I'm managing within Kubernetes as well as the actual versioning of those applications and their states.

Absolutely. So historically, Git has only been used for applications. What we're doing is extending Git through operations. So you're using Git for both applications and operations. That's spot on. We've been using Git on the operations side for a little bit of time, probably for I think – and I agree with you it's growing, but I probably got exposed to it probably about four years ago at scale of where we're using Git to say I need to answer this larger question of how do I show the auditors that I'm maintaining my infrastructure in a compliant way.

So as I think of infrastructure as code, I have to change my operating model of how I do change management, and Git just happened to be a really great design to do that with. So us folks on the operations side, we've been kind of aware of this Git process, and we've been leveraging it as we're answering questions from auditors, showing that we still have proper change management as we get away from managing individual states of a router. We're managing the state of our infrastructure, which a subset is that router configuration, and that's in our Git repository.

Yep. So I see a natural convergence here. I'm already mature, and I'm using this GitOps style of operational management. What I hear you're saying, successful organizations apply those same techniques to Kubernetes, and once you apply those same techniques to Kubernetes, now you can leverage a consistent LinkedIn – NICRD distribution across multiple clusters, this whole distributed system that you're giving developers the ability to create. You can provide control to, so a developer, when they're developing a new application, don't have to deploy a whole new instance of LinkedIn to do service mesh and debugging.

They can just use my central instance, which I'm controlling and configuring via this GitOps methodology. Absolutely. Go directly to your source code repository, whether it be GitLab, GitHub, Bitbucket. Go right to Git and manage it there. So let's talk about what you've seen kind of in the field. Let's start from the bed. What have people done wrong when they went off onto their Kubernetes journey? I don't think that anyone – you can say wrong, right? We have early movers.

You look at Fidelity. They took a huge plunge into AWS EKS. You can see us on stage at KubeCon talking about how we helped them deploy that platform in a highly regulated environment. I was also at KubeCon speaking with Deep at New York Times, and they've also been a vocal leader in the cloud-native space. They've done a ton of work in regards to Kubernetes early. I think the situation that they are in today is all of their application teams are running autonomously, right?

So they all manage their own Kubernetes. So operations does not have a consistent way of patching, updating, upgrading, making sure everyone's on the same Kubernetes. So there's a number of operational issues that exist when organizations start moving that fast. However, you can't say that it's bad because they were able to bring immense value to their customers. You look at Chick-fil-A. You look at Cerner. You look at a number of these organizations who have taken a plunge into Kubernetes and been first movers.

You can see the type of progress they've made, but now there's complexity that comes along with it, and what do I do now? So I just spent a week or so before KubeCon at the University of Notre Dame. John Gushman, the CIO there, he called me in because his predecessor was infamous for lifting and shifting everything from the University of Notre Dame data center into AWS. So they have somewhere like 83 percent of their compute in AWS today. However, they didn't touch the applications.

So his conversations with me was, Damani, can you help me evaluate if Kubernetes is one the right solution, how will containerization help me, how can I utilize the best in breed services that AWS has to offer? Like there's a ton of cool things that AWS is doing and these cloud vendors are doing, but our infrastructure kind of looks the same as it did while on prem. So, yay, we got out the data center, but what do I do now? So, again, it goes back to cloud versus cloud native.

I think the cloud native organizations, they move fast. The situation is what it is. Now there's some complexity and they're trying to manage it. And then you have other organizations who did not jump in and they're trying to figure out the strategy. And I think there's help that's needed to architect and design what's right. There's so many decisions, there's so many vendors that are out there in the Kubernetes ecosystem that it's difficult to navigate on your own. So, you need partners, you need platforms to come in to assist to help you make those right decisions.

Because no decision is a bad decision, but is that the right decision for your organization? So, I love that response. So, basically, no, if you're an early adopter, you have all the advantages of early adopters. You also have all the challenges of early adopters. If you haven't gone on a Kubernetes journey yet, you have the advantage of looking back at the early adopters, seeing what challenges that they were able to encounter and overcome or haven't overcome yet. And you're at the, you know, you have that vantage spot of being able to sit back and not make some of those same moves.

And still knowing that we're still very, very early in this project. It's only five, six years old. And it took VMware, I think, 15 years to get to where it's at today. And you can still make very challenging moves when you're moving to a virtualization platform versus a physical infrastructure environment. So, I really appreciate the insight, Damani. Where can folks reach out to you if they want to engage you more in this Kubernetes journey? I'm new to the whole social media.

I joined Weaveworks just about 100 days now. And the marketing team has been behind me to, you know, get out on Twitter. So, I have a new Twitter account. I think you'll find me, Damani Corbin, there. I stayed off of Instagram for a long time, but I had to put that out there as well. I've really been heads down for a long time with customers. And now I was just maybe coming out and, like, sharing some of these experiences.

But Damani Corbin, everything, I think you'll be able to find me. So, Damani, I really appreciate this. If this publishes before the new year, everyone have a happy new year. To find out more about the CTO Advisor, you can follow me on the web at CTO Advisor on Twitter. I've been there since 2007, 2008, I think. I met 55,000 tweets. com. Subscribe to us, like us, share it with your friends. Talk to you next CTO Dose or CTO Advisor podcast.