Modernizing Network Ops in a Post-Cowboy World – CTO Advisor Podcast with Guest Scott Robohn

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

Sounds good. All right, so you're watching a special episode of the CTO Advisor podcast. We are in San Jose. It's the week of GTC. But Scott, we're not at GTC. Not yet. We're GTC-adjacent. We're GTC-adjacent. We are at the un-event or un-conference hosted by both AMD and TensorFlow. Or is it TensorFlow? No, it's TensorFlow something cloud. But the whole point is, it's beyond CUDA, the idea that CUDA is threatening the democratization of AI. So a conversation for another day.

Scott Robon, you are the founder of, I'm going to say, TSE? So Tech Sales Craft is my consulting company. About a couple of years ago, I finally shed corporate employment and decided to do some interesting networking security AI projects on my own. It's been an incredible adventure. It's allowed me to get involved with co-founding an organization called the Network Automation Forum and to start a podcast called Total Network Operations. So we met via a mutual acquaintance within the industry. We followed each other on socials for a couple of years.

Had some fun back and forth on Twitter. Some fun back and forth. Sorry. It's fine, Twitter. Or as I like to say, the Twitters is fine. Sure. But we were just shooting the breeze. And you came up with this story about how you're going to go into a university next week and help them think through modernizing their network operations. Correct. And I had to grab you and say, no, let's just, we're here. Let's have a podcast about the conversation.

So walk me through the big picture of why people are in the need of modernizing network operations. Networking has functioned the way it's functioned for the past 25 years or more the way it has. What's the big changes? Why are they, what's the source of change? So if I were to sum it down to one thing, what I'm trying to do is to elevate the role of operations, particularly in the context of network operations. But you can apply what we're about to say in most IT disciplines.

From a networking perspective, I think about it this way. Architectures have stabilized. The price per port for gear is going down. And more can be done by focusing on what's happening with your approach to network operations and your staff. We usually put a lot of pressure on individual contributors, network engineers, men in the front lines. And we'd say, go study this. Go learn InfiniBand so you know how to interconnect GPUs. You go do this in your own time.

Maybe we'll give you a little budget for training. But it's not just the individuals that need to skill up. Organizations need to look at what new tooling is coming in and figure out how to adapt organizationally in conjunction with having their staff increase their skills. So I think the pushback that I consistently hear from the individual contributors is why. I understand that I need to learn the latest technology around alternate ethernet and being able to network in between GPUs. But it's ethernet.

There's a configuration. There's an idea of managing that config, pushing that config. And I've done this for the past 25 years. And maybe in a non-programmic way, why should I change? You can do better with other tools that are coming to market. And not even just coming to market, but that really come from the software development ecosystem. Network automation is one really concrete instantiation of this. But if you think about moving from the paradigm of I'm going to create that config and I've always written that config and I know how to do it and I don't need accountability anybody else, well, that's not how developers write code.

And they'll have a work breakdown structure and they'll have pieces of the code that need to be checked and independently tested with a problem that's divvied up and put into usually a Git repo with lots of quality controls and tests done before things are pushed into production. So I think it's impossible to have this conversation without talking about AI. Because there's network automation tooling. Sure. And then there's the process of being able to get to the end state, which the configuration, your intended configuration.

You can use solutions like Apstra, EDA, et cetera, to do a config. Correct. Or you can use AI to do a config. Kind of abstract the way for me, when am I using one approach or both approaches to get to that end result of basically getting more from a CTO's and a CIO's perspective more out of my team using these tools? So I'll add one other approach in there that really comes down to templatizing. Stop making everything a snowflake. Within your enterprise network, you might have a branch config.

And there should be 80% to 90% of that. It would be standard. Maybe there's small, medium, and large. And there are certain things that are a little different about small, medium, and large. But 80% of it is common. Your core network, that should still be 80% to 90% common. Find the low-hanging fruit where you can systematize things. And those are the things you can easily automate. So if I'm new to networking, I might find that a little odd.

Because this is something we do in application development. We have microservices. It's something that we do in server administration. We have golden templates or containers. We have some type of minimum common base that we then build upon from that. How is networking not caught up with the rest of enterprise IT? That's a great question. And I couldn't have paid you more to tee that up. We did not coordinate on this. There's a really, to invoke your hat, a very cowboy culture that has started out with networking that's more than just configs.

Like, I needed to know what makes this T1 work or what makes this ethernet link work. And it's the combination of the physics of telecommunications infrastructure combined with these specialized computers called routers, switches, and firewalls, and all sorts of other different types of network elements. But with the tremendous growth of the internet, like even going back into the 80s through the 90s into the aughts, you had a lot of, I want to be the cowboy. I want to be the person that gets this done.

And that created a culture that still has momentum to this day. And you're smiling, so you know a little bit of what I mean. Well, you're hearkening back to my Cisco studies when I was learning about voice over IP, ironically, which is something that we all take for granted today. Sure. But learning the basics of tip and ring and why and how I'm translating telephony into networking and the importance of knowing the basics so that I can troubleshoot poor voice quality.

Those two things had to go hand in hand. But you look at modern troubleshooting tools and modern implementations of IP telephony, I don't know if people really troubleshoot at that low, low level unless you're doing it at a high scale or at a need-be situation. So I absolutely can relate to the cowboy mentality. That's right. Not just because I have the hat on. I didn't ask you to wear the hat. This worked out perfectly. Great, great illustration material.

So what I don't get is that managers, leaders have been trying to transform for years. Where are you seeing successful transformations? So I think there are a lot of folks in network engineering that love to learn new things. And I call it, what does a next-gen network engineer need to look like? And I think the next-gen network engineer, which really is the generation now, not the next generation, needs to be much more software literate. And I want to be really careful I'm not just blanket applying DevOps to networking.

Right. Because doing A-B testing of network gear is really hard. It's hard. Configuration. And not everything maps 100%. To development. But if I look at what's happened, there have been disruptions in computing. You and I may or may not have written programs on punch cards at one point in our past. There's been a lot of progress and evolution in computing. Every industry sees disruption. The network engineer can't assume that they'll never see disruption too. But the positive side of that disruption is there's all this tooling that's evolved for software development, like GitHub for code repo, automation code repo, and configs.

There are pipelines where I can actually set up a real workflow to do dev, test, and production, where production is the implementation on physical routers, switches, firewalls. And all the things that come from the software world, like Mark Andreessen told us, right? Software eats the world. I know what he means when he says that. He's not wrong. But all software runs on hardware. So there's that need for skilling up and using the tools that are much more commonly supported.

But there's also a responsibility in organizations to look at, well, how are you managing things? How are you supporting your employees? With new tooling, should I be doing the same old thing with old busted processes for all these new tools? I don't think so. So it's a two-prong approach. Help individual engineers become next-gen network engineers, and help organizations adapt to what does it mean to implement a CI-CD pipeline for network configuration, even if we don't call it exactly that. So as network managers and as VPs or directors of infrastructure, what's my first move in helping to modernize not my network infrastructure per se, but my operations for infrastructure?

How do I prepare my team for the journey that we're obviously all getting pushed into? Well, I'll sum it up in one phrase. Operations teams need to participate in their own rescue. So there's often a dynamic of there's an architecture layer that sits right below the business layer, where they have the relationships with vendors. They get the attention in the steak dinners or whatever dinners that they like. The architecture team hands things off to the engineering team for implementation, and the ops team gets stuck with, this is what you have to do to operate it without any real structural thought into architecture and design for operability.

I'm very happy I'm at the steak dinner portion of my career, but can do it you and me both. Yes. Ops teams can't wait for that to change overnight. They need to take small steps. And I use participate in your own rescue pejoratively, but they need to set aside time to evaluate new tools and tech that are coming out from an operations perspective and not just have it handed to them. Be able to know what's going on in the industry, have a pulse on what's coming for operations purposes.

So attending events like this is helpful. Definitely part of the puzzle for sure. And then we'll give a plug to you help run an event network automation forum. That's correct. Next event? Coming up the end of May in Prague, 2025 of May. Might have to dust off my passport. You're welcome. There's a seat for you, Keith. I appreciate it, Scott. And tell me about the podcast. So Total Network Operations sits on the Packet Pushers platform, where we've been going since August 2024.

It's a great combination of talking to network operators to help shine light on good things they're doing and to technology providers to help build their sensitivity and help them tune into what operators really need. Yeah, I'm a big fan of the Packet Pushers network. I'm still a little bit sad that Greg has retired. But great staple of talent, as you just see. Go ahead and follow the link below for more Packet Pusher goodness. You want to learn more about network automation forum, there is a link in the description below.

You want to learn more about the future improved, the future improved dot com. You can follow me on the web at CTO Advisor on most platforms. I'm starting to be a little bit more on blue sky, but X is still the primary platform in LinkedIn. Talk to you next. CTO Advisor, CTO podcast.