Continuous integration networking - CTO032

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

So Matt, introduce yourself for the CTO advisor audience. Yeah. So yeah, my name is Matt Oswalt. I'm on Twitter as Mirdin, M-I-E-R-D-I-N, and you spent basically the last four or five years doing some consulting for network infrastructure and recently moved into where I started out in the industry, which is a software developer. So I'm returning to my roots. That's a weird progression going from a software developer to a network guy. How did that happen? Actually, of the transitions that I've gone through, that's probably the simplest to explain.

Essentially, I was working as a software developer for a large retail company doing some voice over IP applications, and we sat really close to the network infrastructure guys, network architects were really close, and we would always have meetings with them. This was back when voice over IP was really getting popular. So naturally, we would have meetings with them about how our two teams should interact, and I was in attendance. I didn't really know much about voice over IP or networking at the time, but I was very interested in it, and it was one of those things where I was just really, I guess, intrigued of the power that the network guys had.

So I just got involved very gradually that way. I got my CCNA, left that job to go do some consulting for a startup back in, this is when I lived in Cincinnati, Ohio, and then just moved up, moved to a different VAR, sort of progressed along the certification path. That was like the first transition that I did, and at that time, I was sort of, I don't think disillusioned is the right way, but I wasn't sure if software development was what I wanted.

What I'd seen thus far didn't sort of grab me, certainly not the way that software development's grabbed me lately. So that first transition was a little different, I think. So you went back to kind of your roots, going from a developer to networking, which is a really interesting thing, because we talk about these things, and I don't use the term unicorn often on my show, but you're a unicorn. You know, in this virtualization world speak, you're a networking guy that completely understands and gets development process.

Do you run into many other full-stack engineers in your day-to-day? Well, in my day-to-day, I work on a team of other software developers within the network infrastructure context, so the answer to your question is probably yes. But I would certainly be willing to say that, on the whole, as an industry, we still have a long way to go. I'm certainly not suggesting that everybody becomes software developers, but I do agree with, I think, what you originally said, which is that an understanding of more than just simply configuring network devices is needed, right?

We, more and more, and this is sort of one of the reasons I made this part of the transition, was more and more, actually, network functions are moving out of the hardware. They're moving into, especially the simple stuff, the simple stuff that doesn't necessarily need high performance, maybe just simple network services. These things are moving into software very easily, actually. And so it's not necessarily that network engineers need to learn to become software developers, but by and large, the existing software developers, especially infrastructure-focused developers, are starting to take on more and more of that pie.

And so I think it's worth picking up a few skillsets, just so you can at least be aware of it, and be aware of some of the additional tooling that's there. So one of the really great things about your crossover, and I've been following this since we met at Interop last year, has been, you're taking disciplines that you've learned from your software development career and start to apply that to the networking field. And one of that is the rigor of test, well, promoting and testing code.

One of the struggles that we've had over on the network side for as long as I've dealt with networking has been this ability to test changes before we go into production. Sure. And this is something that you've been pretty passionate about for quite some time. Can you speak to one that the challenges that you were looking to kind of solve? Yeah, for sure. I mean, like if you look at what we're doing with network automation today, we've actually done quite a bit.

And I mentioned, I think, on a previous podcast I was on, I think it was Scott Lowe's podcast, where I basically said, we're sort of making this up as we go along. Network engineers, a lot of us grew up, maybe went through a CS degree, and we know how to write code to a degree, but by and large, we've been focused so much on network infrastructure, we don't necessarily know like a lot of the methodologies and the discipline that's been built up around that.

So could we write a script? Absolutely. But because software development isn't our day-to-day, we miss out on all of the really important stuff. Writing code and having that as a skillset is important, but I think another really, I would say more important skillset is to think about the context of your changes and think about what kind of things you could do to do these things more quickly and more safely. Software development has come up with a whole bunch of different methodologies that promote these kinds of things, one of which is continuous integration, wherein you essentially make changes at any time of the day.

It doesn't matter whether or not you're undergoing peak utilization on your network or your applications, it just doesn't matter because we've defined these use cases in code. We've defined these unit tests and integration tests and so on that really emulate actual user interaction or maybe not necessarily user interaction, but just system interaction. We've built these tests ahead of time, and every change that we make, we subject to this gamut of tests, and that gives us, I think, a little better confidence. Nothing's perfect, it's never gonna be 100%, but it's definitely a lot better than nothing.

It's definitely a lot more confidence that we can make these changes during the day, and right now in networking, we just don't have that. We don't have that confidence that we can make changes at any time of the day. m. m. maintenance window as, frankly, a corner case, right? m. when the network isn't anything like it is when it's actually being used? And so I think that's just a little bit silly. We need to be able to, you know, we've come a long way in network automation.

We're doing things that I was advocating for, I think, two years ago, and that's super awesome, things like templating our changes, making sure that our changes are consistent. I think, by and large, people are starting to catch on to the fact that network automation really isn't about speed. It's more about consistency, and it's about safety, and keeping the network up as a result of that, but there's a huge piece that we're still missing, and that is this, okay, what defines success? , what is it that we do that we verify that we've done everything successfully?

And I mean really successfully, not, you know, successful enough to go home. I mean, like, we're not gonna get called tomorrow successfully. That's a big deal, and like I said, we don't really have an analog for that today in networking. And so there's, I think, two distinct challenges, at the minimum, when I think about this whole ideal of being able to test changes prior to the implementation, or during the implementation, whatever the process we settle on ultimately. One, when we think about automation, automation is a necessity for the next-gen data center, whether you believe in the view of cloud, hybrid cloud, all of that stuff is irrelevant.

From a continuous operation perspective, you know, I myself work in an environment that there is no good time to make a change. , doesn't matter if it's Saturday or Sunday, we're a 24 by seven manufacturing shop, so someone's going to complain if we don't get the change right. Then two, there's the speed of business. You know, we're in a continuous operation shop, and then we have changes, or competition rather, from shadow IT in the form of these public cloud providers that can basically cut through the red tape and give end users what they need.

com, and I'll link to it in the show notes, is, you know, the example of the templating, and the junior engineer who needs to make a change, and he has to go to a CLE and make that change, and the reason why we do changes at 3 o'clock in the morning is because we didn't trust the junior engineer to consistently make that change throughout the day. So those are real problems. I'm going to throw you a softball here. Wouldn't it be great if we had some type of open source solution that would help us solve that problem?

Yes, it would. It would be fantastic if we had that. Yeah. Do you know of any projects like that? I do. Before I get there, you know, I sort of just want to build into that real quick, and I think you're onto something there. This is, you know, this is, you're never going to have a silver bullet for this, right? No, no. This is absolutely not a technology problem at the end of the day. I very strongly believe that.

There are technology solutions to this, but they're never going to cover the whole problem, right? So you need to be able to solve this problem first with understanding what the, you know, the value of consistency to your business, essentially. What is it that you're trying to achieve here? We've sort of entrusted our, you know, the other areas of our IT shops to, you know, these automated solutions, but right now we're still, you know, I make the analogy, you know, basically the way that we're treating network configurations, if you let somebody, if you let somebody, you know, mess with the HTML source of your, you know, business app, you know, live, just changing it on the web server, that's essentially what we're doing in network infrastructure today.

And so, and so first and foremost, I think this is a culture problem. That said, I do want to talk real briefly about a project that I've been working on the, that tries to solve, I think, a little bit of this. And basically the problem is, let's say, you know, you want to be able to run these tests. m. change window, where we've gone through the process of, I don't know, upgrading a few switches operating system. That's a pretty simple and very common change, right?

And you might have a bug in the OS, such that, you know, if subjected to a large amount of traffic, you know, it'll start dropping a lot of frames and you'll have a very poor user experience. m. is not peak usage. So you're not going to be able to see that until you arrive in the morning. You want to be able to subject your network to the same level of, same level of testing, or the testing needs to be at the same level of a real world use case, right?

You know, network engineers, at the end of a change, you tend to like, you know, ping, you know, a web server or maybe the internet. And the reality is your customers aren't pinging you, right? They're using your services, they're buying things on your site. And so your testing needs to reflect that. So, to that end, I have been working on a project for, I think a little over a year now, called TOD. And that stands for Testing on Demand Distributed.

The idea here is that I want to be able to do two things. First off, I want you to be able to run tests on your network infrastructure, and not just on your network infrastructure. You could be doing things against, you know, a web server. The endpoint is really irrelevant, right? The endpoint can be whatever you want. You could be testing the network infrastructure, or maybe another endpoint. That endpoint may or may not be under your control. It really just depends on how you want to define it.

I want you to be able to do that in a very distributed way, right? I want you to be able to increase the power of that test, just simply by spinning up more agents. We have the concept of agents in TOD. And so it's very scale out in that way. The other thing I'd like to be able to allow the users to do is define their own tests in a very generic way. I don't want to write, you know, maybe a test that allows users to do ping tests, or iPerf tests, or anything like that.

I want to define a very generic interface, and then build apps for that, right? So there's this concept called testlets in TOD that allow you to define the basics of an application, a testing application, such as what arguments does it take? What kind of output is it going to give me? And then TOD just distributes that. TOD allows you to take that one little, you know, module, and distribute that across your network infrastructure to really increase the power of that test.

So that sounds a lot like a solution like what Spirit would offer, and the ability to do that with a product that, what they're, you know, they have this new as a service-based solution that we saw at Virtualization Field Day some months ago. Is that a similar approach, or how is it different from something like a Spirit? Yeah, so actually in TOD, I actually took some inspiration from a lot of other projects, right? The common, you know, being able to do these tests in a distributed fashion, certainly there's some similarities there with existing closed-source projects, but I think one of the things that I like to focus on for this, especially, and this is really, really important considering that I've open-sourced it under the Apache V2 license, by the way, is that the flexibility you get from being able to define your own tests, right?

Most of these vendors that offer these distributed application testing platforms, they tend to be sort of limited, right? They tend to be sort of limited to maybe HTTP tests, which is super valuable. I don't wanna, I certainly don't wanna say that that's not useful. It certainly is, but they tend to define like a dropdown of apps that you can test, and sometimes that's useful, and sometimes it's not. Sometimes you need more. So I just, one of the things that I did with TOD is just develop this super generic interface that allows you to define literally any test whatsoever.

You know, the concept of a testlet is just that. It's that generic testing application. You could literally take the ping binary and distribute that. You could also define your own. Maybe you have a certain business app that does, you know, write operations against a database. You could package that as a testlet and then deploy that in mass, right? It really just doesn't matter. From TOD's perspective, TOD doesn't care about the application. So a couple of things. One, I'm impressed that you decided to open source this idea because it sounds like a million dollar idea.

You've obviously spent quite some time on it, and I know you're passionate about getting this out there and helping getting some additional eyeballs on it and getting it expanded fairly quickly. Hopefully we'll talk about that briefly. But another piece of it, which is the thing that I find interesting, since you're a developer first and you're formal training and a networking guy. Sure. I haven't taken a look at TOD yet. From a network, I mean, from a user experience, what is the user experience like from a network engineer looking to use this tool?

Yeah. Well, that's another area where I think, you know, TOD's a little bit different. You know, a lot of the tools that are out there focus on maybe one particular use case, and I didn't want to constrain the use case there. I wanted to make sure that I answered at least a few different possible perspectives. One of which is the thing I alluded to before, and I really strongly believe that, you know, we as an industry need to get better at being able to automate the testing and the verification of our changes, especially when it comes to network infrastructure.

Certainly not limited to network infrastructure, but right now network infrastructure is, I would say, one of the areas that is in most need of this kind of thing. You know, we need to get better about not doing these manual bespoke changes to our network, and this includes verification of changes. You know, automation is great, but without verification, it's kind of pointless, and it's actually quite dangerous. So I think in terms of a use case for TOD, I really strongly believe that a really powerful use case for TOD is to just simply make it one of many tools that you embed in this, you know, let's call it a continuous integration pipeline for networking, right?

So the same software, the same tooling, the same process that software developers have been using for years to push their code into production at any time of the day, we can reuse a lot of that. Like, there's a lot of that that's reusable from a network infrastructure perspective. Even if you're not talking about writing source code, this could be a simple matter of making config changes in a testable way, right? So this is where I was saying, you know, this isn't necessarily a technology problem.

I would classify this under under process more than anything else. And so with TOD, you certainly could say, hey, you know, I want to run a test, and I want to be able to run, you know, bandwidth tests between all of my different sites before and after every change. And then you can have some sort of post-test analytics that says, oh, well, hey, you know, I ran the bandwidth, you know, before the change, and it looks like there's a pretty significant dip after the change was made.

We might want to look into that. That's something that we currently don't really do, or at least we don't enforce. We might do that sort of manually, but it's not something that we do in a pretty consistent basis, I would say. So when we talk about process, and this is not just, I think when we look beyond just networking, or network changes and network testing, this isn't limiting the scope of what you're looking to do isn't limited to network testing. So let's say, you know, I'm SAP guy or SAP infrastructure guy, and I'm looking to roll out a new SAP appliance, HANA appliance, and I want to test the performance of transactional, I want to, you know, test a new transactional application.

I can not only do my traditional SAP HANA load-based testing, but I can also leverage something like Todd to see the impact on the network and get a little bit more granular view. And I can come up with a complete package, decide, hey, you know what, if I'm going to do, and then later at a later date, I can say, you know what, if I'm going to make this specific application network change, or network change, I can know exactly, or much more specifically today, the impact of that change, because I've included this Todd module in my testing process itself.

Yeah, that's the general idea. Now I would caveat that and say like, you know, if you're talking about software development and continuous integration, right, you know, integration tests and unit tests are awesome, but they're not infallible, right? At the end of the day, you still have to define a proper test. So it's still, you know, again, that's why I said earlier, it's not a magic bullet, but it is something that you can use to get a lot more confidence in that process.

And to that point of process, you can have a very rigorous testing process that you just practically don't have the tools to implement. You can have, you know, you can on paper draw a very rigorous testing process, but just simply not have the technical capability to execute it. And I think that's been the case many of times in network. Sure. Yeah, I mean, we're still really addicted to doing things manually on our own laptops instead of building it into a repeatable pipeline.

That's why I've championed, you know, for the last year, that's why I basically can't shut up about the concept of continuous integration, right? Developers aren't, you know, when they push source code into production, they're not actually going into a server and running their tests manually. It's part of a prebuilt pipeline. We have this concept of release engineers who their entire job is to take changes and make sure that they go through this rigorous set of tests instead of, you know, developers pushing code directly to production, right?

You know, like FTP, you know, as dramatic, you know, we laugh about that, but that's actually what we're doing. That is what we do on a network side. On a daily basis. And even on a server side a little bit, that's exactly what we do. Yeah, for sure. And then I do want to come back to the use case part because I, you know, we talk about this continuous integration stuff and that's great. I definitely think that that's one of the biggest things here, but just so that the network engineers on the call haven't hung up yet, I would say also the, you know, Todd comes with a CLI client.

You absolutely can run these tests in an ad hoc manner. I've provided quite a few tools for you to be able to do that. So if it's a simple matter of, you know, hey, all this continuous integration stuff is great, but ultimately I just want to run, you know, an HTTP test between point A and point B, and I want to see the results like right now, that's something you can do with Todd as well. Both are supported. So let's end the conversation on two things.

One, the open source project itself. I noticed a tweet from you the other day that said that you already have some commits from someone other than yourself. So the project obviously has some interest outside of yourself. Yeah. Yeah, yeah, yeah. There's a few folks that have told me that they plan on contributing, you know, here in the next few weeks. I've been monitoring the graph on GitHub to see, you know, who's forked it and what commits they've made. And so this is, I think we're not even a week after it.

It will be tomorrow that it will be a week after I've announced this thing. So it's pretty interesting. This is the first time I've done anything like this. So it's, you know, maybe this is small potatoes to somebody else, but it's pretty fun to me. I don't know if people listening to the podcast can get it, but this is something that you're very excited about. You're talking to Interop on a related topic. What's the Interop session going to be about?

Yeah, so essentially, you know, the first talk that I gave on Todd, this was a sort of a really awesome, but relatively, you know, short event for, you know, basically called DevOps for Networking. I think I got that wrong. Maybe it's DevOps Networking Forum. This was a sister, a mini sister conference right before ONS, Open Networking Summit. So I, you know, really only had 30 minutes to do all of this and to demo Todd. I did a live demo, very brief.

So what I hope to do at Interop is two things. First off, I'd like to do more, just period. I'd like to just do more about what I wanted to show, but I also want to show maybe some new functionality and definitely dive a lot more into the process, the part that I've talked about here with you. I definitely want to spend more time on level setting. Why, like, why are we doing all this? Todd, you can't just, you know, make install and now you have Todd and now you solved the problem.

Certainly not, right? This is definitely part of a much bigger picture. And we'll get a clearer picture at Interop. So I have to, as I do every podcast and I talk about Interop, I give a disclaimer that I'm the application track chair for Interop. And I invited Matt to come through and talk about when Todd was still just an idea and he was still banging around with it in his, I guess, ample spare time, because this is not a small feat to release this project.

No, and in fact, I do have a day job. This is not associated with that. So I definitely have responsibilities there as well. I don't have ample spare time. The, you know, computer science 101, right? Everything's about trade-offs. The trade-off for me was sleep this time and I don't recommend it. So where can people find, because not all of us are GitHub mavens. If we wanted to just go to it and get it and download it and play around with it, where do we get started?

Yeah, the best thing to do for that is I tried to do as good a job as I could. And certainly I can improve this, but I've provided a nice walkthrough for getting Todd started in a Vagrant machine. So if you have Vagrant installed, you really just need to run a few commands and you have a working Todd implementation. com slash mirdin, M-I-E-R-D-I-N forward slash Todd, T-O-D-D, there's a nice readme there and you can follow that to get it started. com forward slash podcast.

You can find the notes to all of the links and things including Interop that we've talked about in today's podcast. So Matt, let's wrap it up. And once again, where can folks find you online, blog, Twitter, all of the above? Sure, my two, definitely my two most used and most watched places for that social interaction would be Twitter for sure, mirdin at M-I-E-R-D-I-N. And then I try to blog as much as possible. It's sort of decreased in the past half year or even year or so just because I've done a lot more.

Yeah, I think there's something to be said for writing code versus writing blog posts. The more I write code, I think the less I blog just because it's sort of scratching the same itch. You're right. net. And I look forward to seeing you at Interop in a couple of little bit less than a little bit over a month. Yeah, it's coming up. It's coming up quickly. Again, thanks folks. com. Share it with your friends, rate us on iTunes and come listen back, subscribe.