NetOps vs. Cloud Native

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

All right, we're going to try not to get super geeky on this conversation, but it's hard because I have Ed Warnke on. Ed, when was the last time you were on a podcast? I don't think I was. Oh, that's right. You were doing it live at some event and we were talking about the FIDOVPP stuff and Terabit soft switches and that kind of stuff. Super geeky conversation. I'm going to reference it. It was all about cloud-native networking. Terabit has decades of experience in there.

He's chaired the Service Mesh Project. What part of the Service Mesh Project at CNCF? Oh, so basically I am one of the co-founders of the Network Service Mesh Project at CNCF, which is a little different than the rest of the service meshes you're familiar with because we deal with L3, not with L7. Which is really interesting because I think about most of the service mesh problems from an L3 perspective than I do from an L7 perspective. And I had to rejigger my mind to think about this, which is also interesting.

We start out this way because I had no idea that Ed is a distinguished engineer at Cisco. I've only known him from open source networking, which is really, you know, that's kind of cool. Yeah, no, Cisco's been very supportive of sort of doing things in open source that actually move the networking industry forward. And so they've been quite pleased to have me interact across a number of open source communities and really just try and push how we think about the intersection between networking and some of the things that are arising now around, for example, cloud-native, those sorts of things.

They've been very generous in that regard and very supportive. But I'm usually not the guy out there flacking at a particular solution. I'm trying to think about how we solve problems laterally. Pretty cool. So, Ed, I've thought about this topic quite a bit. We have a project going on, a sponsored product project going on with Arista, working with our Cisco 9Ks to do some of this magic, which I think magic is a good way to say it because I've cut my teeth, my networking teeth, in enterprise IT.

And the thought of taking enterprise networking, NetOps, and giving the interface, the API, to my network to some cloud-native DevOps developers scares the bejesus out of me. Like, I just, I can't imagine letting developers, front-end application developers, I'm not talking about network developers, but front-end application developers, make calls to my network to provision stuff in my network. But I can- Congratulations. You've met the minimum bar for sanity because when you look at these things, what you really have to do is recognize everybody's legitimate interests, right?

So the NetOps folks have a legitimate interest in not having the network fall down and die, because it's their head on a platter if that happens. But your developers have a legitimate interest in being able to run really fast and get their needs met, whatever those needs are. And the problem is that the interface between the two right now is usually a ticketing system. And the tickets are just the worst, right? They take a long time to resolve. They've got humans directly in the loop.

And effectively, in some sense, we're misusing the humans because we're trying to get the humans to bring human judgment to bear on a case-by-case basis, where we need to have just pre-thought some of these things a little bit more, right? So when you talk about how is it that you can actually allow developers to get their needs met quickly, well, the first thing you got to do is you draw a nice, tight scope around the kinds of needs they can get met, right? So just as an example, what kind of needs to get met?

But also, you also have to think about how they think about the world and how you can represent the world to them. So we're used to thinking of the deep networking thoughts. We think about prefixes and BGP and Spine and Leaf and all of these things because we've actually got to get the physical world going, right? It's really telling if you look at Kubernetes, for example, where if you look at the Kubernetes API, there are very few places, almost none, very few, that you even see IP addresses.

So most developers aren't even thinking about Layer 3 at all. They don't want to, right? What they really want to be able to do is they want to be able to talk to the things they want to be able to talk to. They want to be able to know that if something goes wrong, they've got observability that lets them figure out how to make it go right again. And then they want to get some variety of security, and the amount of security that people may want is nearly infinite.

Those are sort of the key things that developers want, and they don't actually want to tell you, oh, I need a slash 24 for this, or I need you to reprogram this route for that. That would be insane. So in the open source project that I've been working with, Network Service Mesh, where we're fundamentally not doing the same things that you're doing with stuff like Istio, which is up at Layer 7. We're actually fundamentally living down to about Layer 3. We'll let you do things with Layer 2 if you really insist, but you probably shouldn't.

And the way we conceptualize the world is, say I'm a workload, say I'm a pod or a virtual machine, or hell, even a physical server, right? I would like some kind of, we call them network services, possibly more than one, right? I would like to be able to be connected via something that looks to me like an interface to something that gives me the connectivity, observability, and security that I need for a particular communication with a bunch of other things. So let's pause there for a second.

I want to make sure I paint that picture in my head correctly. If I'm a developer and I'm used to delivering, you know, working at Level 7 and, you know, this application talks to this application, life is good. Sometimes I need to penetrate that. And I need to say this data center talks to this data center specifically. I need to penetrate kind of this abstracted layer of Level 7, and I need to say, you know, either what path I want it to take or whatever, if given multiple options.

Is that the idea? Sort of. I mean, let me give you a really concrete example. Like I'll go really concrete. Let's say that I've got a situation where I've got, I want to do database replication to a bunch of read replicas that are running out in clusters that I have in the public cloud. But the main replica is actually sitting somewhere on-prem, right? And just for the sake of argument, even though I wouldn't advocate necessarily doing this, although it's getting to be more doable, let's say that all these workloads are running in Kubernetes as pods, but they're running in different clusters because they're running in different places.

So I would really like to be able to have each of these databases connect to a network service that I usually will call the database replication network service, which is basically more or less a baseline Layer 3 with a little bit of DNS, maybe, maybe not even a little bit of DNS, where they can speak the bespoke non-HTTP protocols that you're using for that database replication to each other. And literally, that's the only thing going on there, right? Nothing else is happening. That's all that's happening on that particular network service.

And I don't want to think about which data centers things are in. I don't want to think about which public clouds things are in. If somebody else spins up a read replica that I don't even know about, but somebody has blessed off their credentials to be able to talk to the other read replicas, I don't want to have to know what cluster they're in. You know, what I want to have to deal with is the logical problem I'm trying to solve. Yeah, so on paper, this is really cool because as an operator, I can say, you know what, the developer has defined what he wants to do from a replication service perspective, and I can move that replica to anywhere I want that meets the criteria of the developer.

So if, you know, if Google comes up with a better underlay that's going to be faster, et cetera, et cetera, I can move that. But the developer doesn't have to change his application. It's just the interface between the network service and that layer three provider for that. Absolutely. For us, the provider for the network service could be almost anything. It could be your physical network. It could be something running in a VM. It could be something running in a pod.

We're kind of agnostic on that. It could be a bunch of complicated things happening in the background the developer doesn't know about. But so here's the interesting fundamental thing that I just did quietly in the background, though, intellectually, which is I've decoupled the connectivity domain, the domain over which we speak from the runtime domain, the domain on which we run. And this is something that until you see it, it's invisible. And once you see it, you can't unsee it, which is we have historically welded our connectivity domains to our runtime domains, right?

If you're running in your data center, the networking you get in your data center is exactly the networking you get in your data center with some finite set of nerd knobs that maybe you can open a ticket to get tweaked. If I'm running in a VM, right, OpenStack, VMware, whatever, the networking you get in your VM is exactly one flavor of networking. It may have nerd knobs you can tweak, but there's one of them. If I'm running in Kubernetes, I have exactly one flavor of networking I can get.

If you conceptualize each of those things as a network service, Kubernetes networking is just a particular network service that you use for intra-cluster networking within Kubernetes. And the same is true for the networking that you get from your VM. Even you can conceptualize the networking you get from your data center this way. But we've welded them together, and the developer actually doesn't care the question of how do I need to communicate is not really terribly bound for the developer to what cluster I'm running in.

Now, it may be bound to some other topological things like what zone am I in, what geography am I in. Those things may matter, but the fact that I'm running in cluster A instead of cluster B, it's almost a non sequitur. But we've made it central to how they figure out how to communicate, which causes pain. Absolutely. , my application, my database replication is more important than anything. And it needs the service, and I'll play nice if I can, but I just know to get the performance I need, I need to twist these knobs, and I get the performance that I need.

My job here is done, go to the next thing. But I don't consider security, I don't consider bandwidth contention, cost of link, et cetera. Yeah, but the thing is, those are all things that NetApp guys does when he builds out a data center network. Right. Now, but then this is where, like, thinking about the virtual world in ways and drawing selectively from what we've done in the physical world, but not just wholesale slapping a V in front of everything can be very powerful.

So what if I could say, okay, great, you can design sort of a cookie-cutter style of service where, great, I'm going to say, you know, this is a virtual layer three network service, you could have one of your very own by clicking these buttons, these are the knobs that I've decided to give you or not give you about it, and so you can come in self-service for that network service, but within the parameters and the bounds that I have set, right, so that you actually maintain control, a very tight control as a network operator, but you can pick out the small number of conceptual things that developers want and run with that.

That sounds like a lot of work. That sounds like a lot of work, like, I'm taking your network replication service as a possible, as an example. I can't tell you the number of times that I've, that an application team has come to me and said, I have this brand new way to replicate a database, and I'm like, no, I have a finite, I can't, I can support a finite number of ways to replicate a database. Your 18 different flavor just makes no sense.

Well, keep in mind, we're not actually replicating a database, not at all. Right. All we're doing is, you have a way to replicate the database that you've been using since dinosaurs roamed the earth, and it doesn't even talk HTTP, because it was doing that long before HTTP was the one true protocol, but you need to be able to get layer three packets between everybody in a fairly straightforward way, right, you got to get IP to flow, that's all. And so, all we're saying is, we will give you the ability to have a set of workloads, not clusters, but workloads at the individual workload level, admitted with cryptographically secure identities under the policy that you choose into something that will just route those IP packets around those particular set of folks.

Very simple. So, I could have these classified services. So, let's say that I label, you know, generically a network configuration, a layer three and below configuration, so it is ideal for database replication. And it's also ideal for some other similar service. I can service that up as a service that a developer can just name, yep, I need one of those, versus, let's say, a real-time application for a real-time collaboration application that has a completely different look than asynchronous database replication. And I can say, you know what, that's a different type of layer three and below, you can consume, you can ask for this thing, this service, and I've decoupled that from the developer needing to know anything about the underlay or developing for that underlay.

They're just developing for that level of service. Exactly, exactly. Yeah, and then maybe you decide that, you know, things have gotten a little bit hairy, and so you're going to, into that network service, add a little bit of IPS into that network service. You slide that in as the network operator. It becomes part of how the service is defined, and so you can bring the security to bear at the network level without having to go and do sort of all this direct engagement.

You still maintain the implementation and policy control, but you break it into cookie cutter pieces that you can let the developers do self-service-y kinds of behavior starts. So this is great on, I think, Greenfield, as I think about Greenfield and how to move forward. How do I take operations forward? But this is something that I deem most of the cloud-native folks about in general. Most environments aren't 100% cloud-native. I have my HPUX, my mainframe, my VM, and I have all of my legacy applications that I support in this legacy NetOps fashion.

Then I have yet this other model that we're bringing to bear. How do I kind of bridge the two operating models? Because they don't seem to complement one another. Yeah, no, no, that's absolutely true. You and I have both been around since dinosaurs delivered packets village to village, right? Yeah, right. So we sort of get the fact that not only is it, you got to figure out what you're going to do about how you're going to talk to VMs, but you've also got a whole bunch of physical servers that aren't ever going to go away.

And if you're super unlucky, and I hope nobody is this unlucky, you still got freaking IPX wandering around. Oh, I hope to God, no, that would be really bad. Hey, man, we will allow you. Yeah, so effectively, this is part of why when network service mesh was put together architecturally, and I want to be careful not to over promise what we have implemented so far, right? We've primarily done a reference implementation of the open source for Kubernetes. But the architecture is very, very careful to keep an eye on being able to interface with older legacy workloads, VMs, servers, etc.

Because the basic model we have is, if you're a workload, something and Kubernetes, we can hide this because of magic with your missions controllers and whatnot. So it's just an annotation. But something has to make a gRPC call to say, basically, here I am, I would like a virtual wire plumbed into me to connect me to the network service that I've asked for by name, possibly with a collection of key value pair labels. And here's my cryptographically secure identity. The API for it is simple enough, request close monitor, that you can shunt it in front of other things.

And we have adopted what I call a chain of responsibility model. And what I mean by this is that where everything breaks down and where SDN really broke down is when you start trying to string these things together, you start to get delusions of grandeur. You suddenly realize to make this work, you should probably also try and control the firewall, which they're not going to want to let you do, and all these other things. And this is why the SDN model is the sort of overarching tweak all the knobs, which terrifies us so badly.

Network Service Mesh basically takes an attitude, and I'll hum a few bars about how it works in Kubernetes, because it'll give you the idea, that says, when you make this request, the first thing that captures that request as a recipient is some entity whose job it is to get you either out of your immediate environment to where you need to get your rewire strung, or to connect you there. So if I've got a single node in Kubernetes, single server, and I say, I would like the database replication network service.

It first goes to a component on the node whose job it is to either cross-connect you within the same node or get you the hell out of the node, right? That probably involves injecting an interface on your side and negotiating with the next guy down the chain, whatever the tunneling cap type is going to be for this, right? And we're agnostic, we dynamically negotiate in-cap types, because I am so freaking sick of arguing about in-cap types. You've probably been around long enough to also be sick of it.

And then maybe the next hop is, how do I get out of my cluster, right? So the next hop ends up being some entity who figures out how to do the little bit of niceties to make sure you can exit your cluster cleanly with whatever tunnel-y thing you're doing. And that piece, that hop in the chain, can literally be written by somebody completely different than the one, and it doesn't happen to be because it's Kubernetes, but by somebody completely different. And the next step is, how do I get out of my data center, right?

This is for the firewall thing. How do I not break my security by punching through it like a maniac, right? Because the API is so simple, it's a few hundred lines of Go code to have something that catches the API and pokes whatever API you have for your firewall. And this is where it actually makes your life better than what you had before, because every VWire is associated to precisely one workload. So whatever you do to open a micro pinhole in your firewall because you decide you want to let it through, because you're the guy controlling the thing that actually lets you out of the data center, not only is it extraordinarily tightly scoped to exactly the VWire for this particular workload, but every single virtual wire has a finite expiration time by default 10 minutes, unless it's renewed.

So unless that workload continues to be a going concern, that goes away in your firewall at the end of 10 minutes. How many people have you dealt with who have the firewall rule problem where they think it's making them less secure because some of those IPs have been reallocated, but they don't know which rules they can take out, right? And this cleanly solves that. Yeah, I've heard the promise of things solving that, but I think we're getting closer to the reality of it because I'm easily equating the concepts.

And this is what I like about Kubernetes in general. I'm equating the concepts of what you're applying to that underlay and what needs to happen from an API perspective, even from a network fabric. This is why technologies such as EVPN, VXLAN are relevant to people who think cloud-like. This makes these things possible in the data center to all the way to level seven. And what's happening at layer seven is folks like HashiCorp and Glue are thinking about how do I take legacy applications and allow them to talk to cloud-native applications while rewriting the applications?

What is agile and where the development is happening at is at this network service layer that you're talking about, the service mesh for layer three and the service mesh for level seven or layer seven, so that this gooey-ness that has to happen between our legacy infrastructure and new infrastructure can take place and frees us up to think about providing these services, whether they're layer seven services or they're layer three services, we can think about providing them in this abstracted namespace type of manner. Absolutely, because the enterprise applications aren't going to get rewritten for any of this, period, full stop, right?

That's not going to happen. And I mean, hell, it's probably true that a great number of them probably couldn't be if you wanted to. You know, literally, as in we don't touch it, it just works. And if it stops working, we're screwed, because nobody knows exactly what to do about it. So that's absolutely sort of how we think about it. And you sort of mentioned things like the layer seven folks. So network service mesh doesn't even try and play at layer seven, because it's a big space with lots of competent players.

But we're complementary to it, because if you scratch any of those layer seven service meshes, what you discover is they presume that they have under them what they call a flat L3, which for them just basically means there's a nice flat router domain under them, like what you get in a Kubernetes cluster. And this is a big part of why they tend to weld a service mesh instance to a single cluster, and then try and play crazy quilt gateway games to get out of the cluster at layer seven.

The problem with this is that you wind up trying to play a game of layer seven static routes pretty quickly. And if you imagine how bad it is trying to build an entire enterprise network with layer three static routes, it's so much worse trying to do with layer seven static routes. Yeah, I don't even want to try it. No, I don't want to think about it, because the number of applications become kind of infinite. Oh, yeah. No, I don't want to think about it.

But if I can run an application service mesh, like an Istio or Akuma or Linkerd, on top of a flat L3 network service that's being provided, now you can have a single instance and you can stretch it across different locations and simplify that problem a great deal. So, Ed, if people want to go deeper on this, if they want to learn more about the innards of, well, what does it take to actually cobble together network services at this layer three perspective to provide to, as a provider to the operators at the higher level, where can they follow you in the projects you participate in?

So, you can follow me on Twitter at Ed Warnecke, E-D-W-A-R-N-I-C-K-E, if not for the fact that my last name is a little difficult to spell, it would be very simple. io. From there, you've got links to the Twitters and the GitHub and all the rest of the stuff and documentation there and whatnot. So, those are the pretty simple jumping off points. All right. And if you want to find more about the CTO Advisor, we'll tease a future podcast with Ed talking about GitBomb, which is, I've just took another look at it as we were talking, he gave me a little bit before we recorded.

But if you're the least bit interested in software supply chain, this is going to be a great conversation in a future. Ed, I'll schedule some time to record that with you. com, forward slash podcast is where you find the podcast. We also have a blog and other video content. If you want to learn more about me, you can follow me on Twitter at CTO Advisor. DMs are open so that if Ed is just too geeky for you and you want a dumbed down version of it, you can DM me.

Talk to you next, CTO Advisor podcast.