The Edge Control Plane - Distributed vs. Cloud

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

All right, you're listening to another episode of the CTO Advisor. We're in the week of VMware's acquisition and we're not going to talk about the VMware acquisition on the today's podcast. There's been plenty of commentary on that. We're going to talk about the edge in the edge control plane. I have a industry technology strategist, Sarbjit Johal, who has been on the project podcast before. And then I also have Rob Hirschfield, CEO of Rackn, who's all about the edge. And we're going to talk about the edge control plane.

First off, guys, welcome to the show. Thanks for having us. All right. So we're going to level set. We're going to start out with a simple question, but loaded, Rob. What is the edge? I'm waiting for the simple question, Keith. The in a lot of ways, I think that edge is best defined very broadly. You know, in a lot of cases, it comes back to times when you're building compute infrastructure that is in the physical world. It's related to things that are happening in the environment.

And it can be as simple as what's going with the cell phone, the cell phone that you have. It could be as complex as a factory with a whole bunch of automation and control systems that are local to the factory and have to build tight control loops. But fundamentally, when we talk about edge, we're talking about computing systems that interact with the local environment more. And then and there's another important thing, because there's an element of every everything edge is not cloud.

And that definition is useful because one of the things that makes edge distinctive is its resource constraint. That might be a latency or a networking constraint. It might be a storage or compute constraint. But typically, edge environments are much more constrained and limited than cloud environments, which by definition are sort of, you know, entirely open ended and elastic and you don't you don't have any restrictions of consumption in the cloud environment. Edge is the opposite, or at least usefully defined as the opposite in that case.

Hmm. So, Sarbjit, as developers look at potential solutions or coding for the edge, what are you looking for in a platform and what's the expected experience? I think developers look for places where infrastructure is taken care of for them and they they just want to write code. I mean, that's at the highest level and in simplest terms, that's what they want. They want facilities like serverless. They don't want to muck around with virtual machines and even containers. So developers want to write code fast and test that code fast and do quick iterations.

And just to. Elaborate a little bit on the what I my understanding of the edges it is. When developers talk about edge out there, they are talking that in context of IoT and IoT, which is industrial IoT. And I extend that to even people. So those are the things we are connecting things out there. Cars, you know, rockets. You name it. You know, your refrigerator in home. There are things getting connected. Right. But people are important part of that.

I think remote work is a sort of edge location for most people. Even the office is an edge location, because if we notice, even in the last 20, 30 years where we worked versus where our data centers were. Those were different locations. So the workplace is an edge location as well. I think a perfect example of that is, you know, movie studios. When they are producing these animated movies, they muck around with a lot of data. But they are storing rendering actually most of the movies in cloud now.

So that's an edge location as well. The doctors and hospitals is another example, for example. So I think coming back to developers, developers want to write code and test that fast enough. That's what their dream goal is. And they don't want to learn new skills to do that. Let me add that they want to leverage what they already know, the languages, the frameworks and and APIs they already know. So that's another aspect of it, I think. So for those listening to the podcast for the first time, we had a conversation that should have been published a week before this around WebAssembly and this concept of using WebAssembly at the browser to deliver Photoshop.

We had an engineer from Adobe talk about WebAssembly in the edge and developers being able to use C++ to run code in a web browser. So subject, I absolutely agree with you. Developers don't want to learn new skills. They know how to solve problems the way that they know how to solve problems. But both of you brought up a common thread, which is the edge has everyone has finite resources, even a public cloud. But in a scale of things, the public cloud resources seem infinite versus the if I'm a movie studio, I only have so much local capacity.

I have so much bandwidth to move that stuff from the edge to the data center to be processed. And I have other limitations, only have so much CPU, so much memory, etc. So, Rob, you you're in the super low level of this. You're dealing with everything from Intel nooks and smaller up to server class equipment out in all of these edge locations. What type of applications are you seeing being deployed at the edge? Oh, boy, the the challenge with edge right now is that people people need to realize edge is something we've been doing for a long, long time.

T. principles coming to the edge from that perspective. And I think it's right that we're also going to see cloud based development processes and platforms coming to the edge because developers are used to working in those platforms, which. So the thing that I'm trying to put a fine a fine point on here, especially because we're talking about control planes ultimately, is that the edge has already been around. We've been doing operate IOT control systems, right? All these things we've been typically doing it as what people in the industry will call an OT play and operational technology, meaning people buy an application that does one thing, security cameras.

And they buy they build a security camera infrastructure that is just a security camera infrastructure and it is a dedicated application for that is built as an application for that. And then they'll buy a cash register system. That's a cash register system. T. system. They manage them as the cash register team manages that. T. T. manage it. In some cases, because we're now connecting these things to the cloud and you have to have the integrations between these operational technologies, these OT systems back into all the back office stuff, which is running in data centers or in the cloud.

T. and developers and application platforms, right? We're starting to, you know, in some ways make the edge disappear from a separate environment. It needs to become a connected environment. Yeah, from Rack End's perspective, that's the story that we hear people saying is like they've got these applications. They've got they want to say, all right, I'm going to consolidate my cash register system and my vision system to both use Kubernetes in the back end. And maybe I should use a Kubernetes platform.

So this intersection of, you know, the cloud development platforms and the way software is being developed for that is now coming into these environments that used to just be standalone silos. And we're looking at building shared infrastructure for them. Yeah, that's the that's the transformation. Yeah, I love that breakdown because it clarifies for me when we first started talking about the edge as a market capability. I'm thinking, you know what, we have the edge 30 years ago when, you know, I was helping, you know, the local business with their Tandy 1000 base thing.

It was, you know, it was an OT thing, a serial port connected to a Tandy 1000 collecting data off an OT system. The system that's, you know, that's isn't that the edge? But I think this this the expected for this conversation is clarification that edge managed in the IT fashion is really what we're talking about, because there's potential there. So, Rob, who's leading these efforts and how are they being frustrated? Because when I think of the capability, I don't necessarily think of IT.

No, and this is why I think the control plane pieces is really is really hard. And one of the things that you said made me think back to, like, the colonial pipeline hack not that long ago, where OT systems, right? Logic controllers to manage the pipeline were attack vector. And so part of, I think, the motivation here is all of these standalone OT systems are vulnerable as IT systems are. And we need to actually manage how all that stuff gets wrapped up.

And so the frustration here is back to how do you actually build an environment that has enough capabilities and operational day to management? We have a day one problem. We have a day zero, day one and day end problem in how you think about this. And so what we've been seeing happen is people build applications for cloud using, say, Kubernetes and cloud storage techniques and cloud networking techniques and cloud security accounts and load balance, like all this stuff. It's amazing and it's easy and you can terraform it up into existence and it's great.

You bring that into an edge environment where those things don't exist. And then all of a sudden you have a very different design pattern that has to be implemented to do that work. And in some cases, the developers are going to be like, oh, OK, I have to deal with a different type of storage or I have to localize my storage. I can't just go to the network. But in a lot of cases, they still need services. And this has always been true.

It's just becoming more and more acute as we've gotten. It's been easier and easier to consume those things. So when you build an environment in these edge systems, there's all of a sudden we go back to who's installing the platform. Right. If you're installing Kubernetes, how are you managing that Kubernetes? Where's the data being stored? How's it being replicated? What's the security model for all those things? If I have a load balancer in front of it or a service mesh or DNS infrastructure, right?

DNS infrastructure in the edge is actually very hard. And then what happens when I have to issue a patch? How do I how do I have that patch distributed to all of my sites? You know, how do I deal with new gear showing up when the gears are like this? All of these operational concerns that we've sort of said, oh, the cloud is going to take care of that and I don't care anymore. Those are questions that that really do make a difference.

And you have to that's what the control plane does. It solves all those back off back end, you know, run the infrastructure, install operating systems, patch, install platforms, patch platforms, provide networking. That's that's where people sort of stub their toes quite a bit. So, Sarbjeet, you've talked about this topic a lot. You're at the forefront of this conversation about making sure we get the edge control plane right. Because of all the reasons that that Rob just shared and the advocacy you're doing for developers, they don't want to worry about load balancers and service meshes and DNS.

They don't want to worry about that in the cloud. And they for sure don't want to worry about that in the edge. So I think we all agree about what the problem is. There's the control plane for IT is heavy to get to the value statement of getting this value out of the data, out of these end user interactions to get that value. The IT model makes sense, but the control plane is heavy. So talk to us about, you know, the idea of a cloud control cloud based control plane for the edge.

And conceptually, what's the difference between that and a distributed control plane? I just want to add a few things to what Rob said. I agree with Rob about what he said about the security and threat surface, you know, is increasing with more devices we put out there, more computing we need everywhere. It just increases the threat surface. Right. And to contain that, I think that's why I want to tie it back to that problem of, you know, pervasiveness of computing out there.

In order to control that, I think consistency is the key. And in order to keep it consistent, I think central places are the best way to control all these devices in the field. So, I mean, I have this developer mindset, like building applications since, like, 95. I think more stuff you can do centrally. I come from all object oriented, you know, or AD world, if you will, object oriented analysis and design. So abstraction is your friend. Cloud gives you that abstraction from complexity.

I think you need to put all the policies in a central place and distribute that to the devices out there in the field. I mean, I think from some points, I think Rob and I agree that there needs to be a central place where you have the core repositories and where you roll out the patches and all that stuff. But I think it's like more day to day kind of stuff, like what happens when something goes wrong? Where do you go to get that?

How do you control the damage when a hack happens? Can you have a reduced functionality? Can you close the ports very quickly if something is happening in some environment? Where do we control that from? I think that's a core function of control plane as well. So keeping all these things in mind, I think central place is much better place. Of course, you will cache what we are cooking centrally on the edge, just in case you don't have the connectivity to the central place, which is cloud in this case.

You still can operate based on the cached policies, if you will. But for the management of these environments sake and for the cost control, how much we spend on people who are managing these things, I think centralization is a necessity. So I love where we're getting to on this. This doesn't sound like a cloud versus distributed perspective. I don't think any of us believe that distributed control plane, meaning the management of the… Rob, correct me on this. Is your position that the control plane shouldn't be centrally managed?

Yes. I think the word managed is tricky here. I think it should be centrally coordinated. And I think that infrastructure as code allows us to have a highly managed environment in a lot of distributed locations, but the locations themselves can be autonomous. I think the autonomy is very important for how the system goes. Now, that doesn't mean that you can't take from a central system or distributed management infrastructure, say, okay, I need to distribute a patch to everything, or I need to roll out an upgrade, or I'm going to watch things happen while I'm connected live.

But the fundamental design for these systems needs to say that edge infrastructure has a completely standalone control plane. Wow, this is really different in philosophy because in one model, I can see the advantages of both models. And, Rob, you're a model. I can chop off a complete, like if I have this application, let's take retail, and I have a series of brands under one company, and I decide to sell a brand. I can sell that brand, and the edge control plane and application goes with that operations.

There's no back-end things to do other than change central secrets and all of that and network connectivity, and that happens. But from a, like, being able to break off that business unit to a different business unit is done versus needing in the cloud model, spin up, you know, somehow segregated and put it into a new container or demark and reassociate control planes and et cetera. However, I love the idea, Sarbjit, of set it and forget it, like not managing everything that we associate with the cloud and what's great with the cloud, having that capability.

But the downside to that is that at some point, the edge has to reconnect with that cloud control plane. It can operate independently for a while, but it has to reconnect at some point. So this isn't a better or worse thing. This is all about, again, requirements. Rob, I'm going to give you the last word. I got to admit, Rob, both me and Sarbjit are at a disadvantage because we don't do this day-to-day, and I have to side with Sarbjit.

I don't want to have to worry about this, but it's hard. You're actually in the trenches doing this, so I think we both have to defer to you. What's the reality? What are you seeing on the ground? Let me give you an example before Rob wraps it up. My door lock is a perfect example. I've used it for more than a year. I love it. It gives me the freedom of opening a door. Somebody comes in from miles away.

I'm sitting in New York. I can open the door at home. That is a great user experience kind of product. Do I worry about the code on that? No. Do I trust the vendor that they are updating it when it needs to be updated because they found a loophole in their operating system on that device? Yes, you bet. I do believe in them. You've got to keep in mind the convergence of technologies. Our networks are stronger.

Our end devices have more compute power than they used to. We are not in this embedded software era anymore. We don't need to just do the control planes locally. I think we can do a lot better by centralizing these. With that, go ahead, Rob. Yeah, I think that what you're describing is not a particularly scalable model for building these systems out because what we need to do, I think where Edge really should be going is driving the cost of adding sensors and infrastructure back down.

And part of being able to do that is to interconnect the systems together. What you just described ends up being an OT silo in your house, and you are, one, beholden to a vendor to maintain all that, which you're hoping they do, and they'll do it better. The common philosophy is that they are going to do it better than you would. And I'm not describing, by the way, technology that is for consumers, although I think consumers would benefit from being able to have a maintainable infrastructure in their house that they then plugged everything into and then didn't rely on sending all the data to a cloud.

But the challenge is if that company decides to go out of business and brick all of your sensors or doesn't make the patches that you think or becomes an attack vector so that you can move through or decides, oh, we're going to share video from our doorbell cams with the police, there's a lot of things that are included in that statement that are out of, to me, a little bit out of scope for the pure what is a distributed control plane and what's the benefit.

To me, what we're really describing here is there's a SaaS model or a service model where you could say, you know what, I don't want to run the infrastructure at my site, at my gas station, at my cell phone tower, wherever, or home, and I'm going to outsource that infrastructure management. That's a perfectly valid model. In doing that model, I would prefer that the company doing that can actually manage, and they actually try to do this, manage all the infrastructure for that site as a standalone site and don't have a dependency back where every control decision, every set up a machine, turn things on, turn things off.

If there's an attack, have to control it from the remote system because it leaves those systems incredibly vulnerable. What I'm suggesting, yeah. No, I love this nuance because, one, I don't have smart locks because I don't trust the vendors that create them. Sure. And I think that is the challenge as a developer slash consumer. If I'm looking at it from a developer perspective and I have an idea and I can build this idea on Apple's home kit without me doing any of the hard work of building a sensor or a thing that reports back, et cetera.

Apple has done all of the hard work for me. Sure. The disadvantage of that is that I'm beholden to Apple. So just in every technology decision, there's a knob. So what I basically see this decision process boil down to is to private versus public, private cloud versus public cloud. And this is the challenge that I've – what you bring up is a challenge that I've debated with the Amazon AWS model is AWS hasn't gotten the bumps and bruises yet to know how to do the edge correctly.

HPE or IBM, Dell, et cetera, have. They understand how to do a distributed edge from a support perspective. , they'll partner with Unisys to do in-home and on-site calls or what was IBM, et cetera, internationally. They know how to get to the edge to manage the edge pieces. As we think of a control plane, we want the same capability, especially if our edge solution scales. So, again, I go back to this isn't an either or the thing. This is what – this goes back to what do you want to build?

It does. One of the things I think that people need to think about here, and this is where infrastructure as code comes in, and one of the things that we're seeing from a distributed control plane perspective that's really interesting. And what's fascinating here is we're talking about this in a framework of edge. We actually have this conversation with customers doing multiple site data centers or having teams collaborating because it's the same problem, right? When you have teams – let me go at it this way.

When you have multiple teams in an organization doing operational work, they typically operate in silos. They're doing similar work on similar infrastructure, but their ability to actually share and collaborate and repeat results is very low. Even the cloud teams, right, operational teams. And what the answer has been and not very successfully is, oh, I'll just create a single control system, you know, and force everybody into a Terraform cloud and make them all one team or an Ansible tower. We're going to force everybody to row together against a centralized control hub.

That does not work well. What I'm advocating here is actually being able to say what we should be able to do is have a lot of smaller control planes, right? So each edge site, going back to edge, has the ability to run its own functions. It doesn't have to be heavyweight. It doesn't have to have, you know, 10 servers to execute redundancy and things like that. We've seen things that you can run a whole site from a switch and put the control plane inside of a switch's capacity to do this work.

But that site can actually have the information it needs to run all the infrastructure at that site in a controlled, consistent, repeatable way. And then you can pick up all of the operational code that runs the control plane and replicate it to another site and another site and another site. We're not compromising your ability to run the site well. What we're looking at doing is actually making it possible for sites to have high degrees of reuse portability in those systems and visibility between them and have a high degree of those control planes being repeatable.

Not custom, not segmented by video and cache register, but back to, oh, I have a control plane that I can build for a site and then replicate that across other sites or into the cloud or onto my colo data center and share the code and share the control plane components. That's where we're going with this. That's where it has to go. As soon as you are doing bespoke site design, I actually agree with you all that bespoke site design is not a scalable model ever, whether it's a team by team thing in an enterprise or down at the edge.

That's the challenge. That's what's been frustrating. So I think we all agree about what the challenge is, is just how do you get to that end result? Sarbjeet, I'm going to give you the last word now because and I'm sorry, audience, we promised that it was going to be typical CTO advisor. But this is a complicated topic and it's really tough to get it in. In 30 minutes, it's a hard problem. I hope you appreciated the extra time we spent to to dig into it.

So, Sarbjeet, I'm going to give you the last word. What are like where do you see the advantages and disadvantages of the cloud control plane? Because one of the things that we all agree upon is when someone else builds it for you, you can typically go faster. And where do you see, Sarbjeet, the opportunity for a cloud centric control plane? Yeah, I think I have made my position very clear for the last three, four years, that the central place is the best place to contain the cost and complexity.

And my question to Rob is that, actually, if you put it that way, is that, okay, you can replicate site to site to site. But if you want to fix a problem on site four, and do you think that only site four has a problem? Do you want to keep it consistent across all sites? In that sort of, once you think about the management of all the sites, I think centralization saves you a lot of grief. , you have different policies.

But I think having centralization to roll out these things, to control the flow of data, to control the policies, because the tax changes every day, tax rates change every day for people who do POSs and all that. You can ask them how complex these things are. Centralization is the key. It's an API driven world. You've got to leverage what's out there. You don't want to do things old way. I can tell you that we're helping customers coordinate and keep sites consistent at global scale.

So, Jeet, I agree with you with the need. We have to be able to distribute updates, changes, and patches and keep sites updated and coordinated, and that needs to be done globally. That doesn't mean that all of each site has to be managed exclusively from a remote location or a central location. We can eliminate the trombone in the control plane. All right, guys. I have so many more questions around this conversation. I mean, I'm thinking, like, what happens, Rob, me and you in the past have talked about, you know, what happens when you standardize on, let's say, an Intel NUC, and to sort of Jeet, your point in the NUC's power supply has changed.

And Site 4 now has two different types of NUCs with two different power supplies. Rob has been dealing with that problem in the trenches, but at the same time, it's really nice for that to be somebody else's problem. So how does that work from a, you know, I'm a company that buys millions of things to I'm a company that buys thousands of things. So, so, so many questions. I think we'll schedule a part two at some point to have kind of that next level of discussion because this is a complicated thing.

There's a lot of opportunity out at, quote, unquote, the edge. But realizing that opportunity, we've given examples of public companies that have started the journey and just stopped the communication. We're looking at you, Chick-fil-A. This stuff is hard, and I appreciate both of you taking the time out to have this discussion. com. Make sure to rate the podcast. I don't say that enough. I don't promote the podcast enough. Rate the podcast in iTunes. If you want to engage with me, I'm at CTO Advisor on Twitter.

Both subjects in I want to call Rob's vehicle because his his Twitter has nothing to do with his name. If you want to find on Twitter, the links are in the show notes. We'll talk to you next. The CTO Advisor podcast.