DevOps or Dev vs. Ops

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

All right, you're listening to Episode insert episode here. We no longer count episode numbers over on the CQ advisor podcast because I don't know really care We have a returning guest last time I talked to Ned Bellavance He was making me pasta homemade pasta and sourdough bread in his kitchen. We're off the road trip We're back at home Recovering from some water damage. So the sound might be a little bit off compared to the regular CTO advisor CTO Those Ned is in his fancy studio Ned how's things going?

They're going great. I'm loving my fancy studio It was a little threatened by flooding as well because we had Tropical Depression Ida come through a couple weeks ago and it dropped I don't know like seven inches of rain on us and fortunately our sump pump Managed to do the job and keep the basement clear, but I was a little concerned for a while there It gets a little touch-and-go there for a while doesn't it it does it does but yeah The sump pump and the battery backup did their job.

So thank you for that. So speaking of concern. We're going to have a Try to have a concise conversation about a really complicated topic which is dev versus ops or Ops versus dev basically a governance conversation There's a whole lot of stuff going on and we don't have a whole lot of people to do that what should developers know Versus what should operations know we're going to pick a specific use case that I got into a really interesting conversation on Twitter about which is replication services Where should replication be managed?

Where should what types of replication should we use? So that we know that we can actually have visibility into the status of replication and fix it when it's broken For the specific application we're looking at We're picking on our friends at port works, which is owned by pure storage. It is a pure software play It has nothing to do with pure storage arrays. It is a born in the Cloud cloud native solution. It allows you to basically connect containers to persistent volumes on any kind of block storage and replicate that to another metro cluster or to long distance for DR and It orchestrates the recovery of your solutions very slick product However, it raises a really interesting question Who's responsible net up?

I'm gonna ask you this who's responsible for replication I mean when I as an enterprise architect, I see that and the first thing that I think of is when it breaks Who fixes it do the developers who implement it fix it do the ops team does the ops team fix it? Who monitors and fixes this? Very good questions and I think we can go even deeper than that because the first thing I think of With replication is what's the goal behind the replication and who needs to understand?

What that replication does? Your replication like you said could be for DR could be because you're running multi-site architecture and you need synchronous replication between sites because sessions are going to toggle between them So the biggest question is what is the replication for and who needs to understand that because they're gonna lay down What I'd call the standards that defines what replication should be available and how it should be implemented Does it need to be synchronous asynchronous? Can it be five minutes behind?

Can it be ten minutes behind? Do you need snapshots of it so you can roll back to a snapshot like lots of questions there I think that's really gonna come down to maybe the governance and compliance teams that sort of layout. Oh, okay Well, this is the data we're protecting and it needs this policy applied to it and also the developers who understand what the actual data structures are that are being protected and from the upside Understanding what's possible with the storage that you have in place today and the network connectivity you have in place today So what you're describing to me is what we you know Have described as a architectural review book board in the past basically if you're going to bring a new technology in We need to Make sure that that technology can be supported.

It meets standards. It's implemented in a way that we That is consistent with our operating model But again, there's something that's sticking in the there's two things that's popping up one that slows things down like the whole thing of cloud native and Well, one of the points of cloud native and adopting cloud technologies is to speed things up To help it helped me understand the argument for slowing things down a bit to be able to Properly support this in the long run Sure, and I think it's a lot of that upfront planning you have to do for any supporting platform or architecture, you know when Say my Microsoft was building Azure and they were determining their replication strategy They didn't hustle that through in a week, right?

They had to take their time and plan out. What types of replication will we support? What SLAs can we guarantee for that replication and then they made that rep those various replication decisions? Available to those who would consume their service or their platform And I think if you are a large enough organization where you have a similar situation You need to make those architectural decisions almost at a platform level and then make choices available to The people consuming your services which are going to be either business units or DevOps teams so let's talk about how something like this is injected into a organization's platform group so we're going to kind of erase the conversation the the concept of dev and ops because platform groups are Complicated they kind of expecting modern platform groups.

They they blur the line between dev and Ops a lot of these folks in the new platform group and we can talk about infrastructure as it as infrastructure to service DevOps Etc, but they have these crossover skills in which they have folks on this team that can call them and they have folks on the team that know the Operations network storage compute virtualization cloud technologies, etc So I guess not erase it. Let's define what is a platform group? Perfect. Yeah, I was actually that's where I was gonna go next is it's it was a term That was relatively new to me until I'd say three or four months ago I hadn't heard it come up a lot and then through a series of Podcasts that I listened to and a few handed up recording.

I kept hearing this term platform team So I decided to dig into it more and it really is exactly what it sounds like. This is a totally separate team and Their responsibility is to provide a platform that other teams can consume So we're not getting rid of DevOps. It's that DevOps is a practice, right? It's not a it's not a team and The practice of DevOps is combining your developers and operations to accelerate software development That's that's kind of the key idea That's still being implemented by each product team or business unit that wants to consume the platform The platform team's job is to build and manage that platform to provide a service catalog to provide SLAs and provide tooling for the various DevOps teams the business unit teams to snap into that and Monitor the status of their application that they're running on the platform and the platform could be a couple different clouds some of that might be Heavily abstracted away or it could be a very loose Abstraction where they know they're going to this particular cloud and he here are the services you're allowed to consume here are the guardrails that the platform team has put in place for you and Here is how you can monitor your application effectively So the platform team is not to get in the way of what the DevOps teams are doing Like those teams still need to own their application and manage the development and operations of that application The platform team is there to enable all the features and services that each business unit needs without While making some Continuated architectural choices for them that support the compliance and security and all those other kind of things that you need in place as a modern organization so I'm going to go back and answer one of my own questions that proposed what began to propose to you, which is why should we slow down and The reason you gave a very eloquent answer in the Azure Example of replication and again, I'm going to give the a similar example in the enterprise when cloud applications first started showing up in the enterprise what would happen would be that the developers who develop the application Needed to go on to the next project in order to go on to the next project they needed to Life cycle the application from active development to steady state operations And when we went to steady state operations that went to a completely different group Which you know back then we can equate to either the platform group or traditional IT operations when that traditionally IT operations group got the request to support the new app They look at it and say wow what what is I don't know any of this like none of this Fits into our support model basically, most enterprise shops are mass service providers at the core of it and they can support a limited set of applications and rather technologies that are built on so if you Have you brought in, you know?

Something based on grafana and and all these new Newish technologies and the application that the support team wasn't ready to accept supporting those technologies Support remained in the development team and that reduced the overall speed to market for developing new Applications because the development team was developing a practice. They were not good at which was Operations So we are getting back to the point of what should the development team know versus the operations team? and I'm gonna stick a pin in that because I want to Get in a point that was very important to the customers I talked to which is a basic question What do I do with my DBA?

So whether you have 10 DBAs 1 DBA 50 DBAs What do I do with the skill set? What do I do with the storage team or the members of the storage team as I think about platform? versus ops versus Developers whether I stick the the talent in and I Don't think we're going to get to answer on this podcast But we can start giving insight into how you think about support and where you put Resources to to enable that support Basic question that you're in the business of Education.

Sure. What are some of the biggest around this topic of? Platform versus devop or platform teams enabling DevOps what are some of the Demands you're seeing from a training perspective. What are people asking to be skilled? The most popular things that I'm seeing from an ops perspective is Kubernetes understanding it and that's that's a pretty large domain all by itself and I don't know if anybody actually understands all of Kubernetes, but Understanding how to operate Kubernetes not develop applications for it, but how to actually operate it monitor it That's a pretty big area of demand and I see a lot of growth there.

Another one is Declarative infrastructure as code How do I first of all write it because I think most operation folks are slowly accepting the fact that they have to do some Programming. So how do I develop infrastructure as code? How do I manage it? And how did I guess that the third big thing is starting to understand? Things like CI CD get ops code repositories how do those all factor into managing and maintaining my infrastructure as code because that's sort of the next logical progression and if you're being integrated into a DevOps style team Where the code for the infrastructure and the code for the application might live in the same repo or adjacent ones Practicing the same software development techniques across both Dev and Ops makes a lot of sense one thing that I want to mention about DevOps in general and the idea of application development and feature development is The best practice if you're trying to do a DevOps style practice within your organization is the team that develops the application is the same team that's responsible for Operating the application.

They don't hand that off to a separate operations team They're responsible for that application throughout the lifetime of that application Until maybe it's ready to be retired and truly is just on life support basically But the idea is if you develop an application, it's probably a revenue generating application And that's the thing you're going to continue to add new features and functionality to so your developers and your operations people work together on that application and I mean the individual team members can continue to cycle through as They want to move on to different applications and projects, but the team itself stays responsible for that Application and might even be in the on-call rotation for that application if something goes wrong Rather than throwing it over to an operations team.

Who's like, I don't even know what a grafana is And I think that that is I think one of the key answers to what do I do with my DBA I think one of the Concepts that we have to start to erase an IT infrastructure that there is a concept or there's this part-line between development and operations And I know as I'm saying this people are thinking about Wow, what about audit? What about? Compliance and making sure that the person who writes the application doesn't have the rights to deploy or change the application Making sure that Security is in place at all levels These are all questions that have to be answered as you're developing your dev ops process It's going to be unique some of it's going to be universal But some of it's going to be unique to your organization and the types of apps you deploy it completely disrupts the traditional stack from a governance perspective you the development the VP of development and We can you know air dirt dirty laundry here.

There's a VP of development and there's the VP of Operations what we're asking to happen is for the VP of development operations either be Put into one role, which is probably not realistic or to have a collaboration between the two in which we have a process The overlaps both teams. So when we ask the question, where do we put the DVA? He may still exist in the same or but now he has a dotted line to somewhere else And I know as I'm saying this everyone is thinking yeah keep we've tried that in the past it doesn't work very well Truth is we have to try new things something's over again because the applications are going to get built and they need to be supported and from a practical perspective to go fast and may enable to be to be able to go fast and Support these applications.

We have to change our governance models alongside Yeah, you'll hear this concept of shift left and the biggest problem with the whole idea behind shift left is moving the security and the operations and infrastructure component to The closer to the beginning of the development life cycle That's often interpreted as the developers now responsible for that Like I'm now expecting my developer to not only know how to write code But also be a security expert and a compliance expert and an infrastructure expert and that's not what shift left means Shift left means you're bringing the folks who are subject matter experts in those different domains Into the process earlier on it's it's that whole go slow to go fast thing at the beginning You go slow.

It's like you wouldn't you wouldn't ask a sprinter to go from Completely cold to running the fastest hundred yards. They've ever run in their life right that they would warm up first Well think of your development process. That's the warm-up period where you're bringing everything online You're getting people warmed up You're bringing in your compliance and your security and your infrastructure folks at the beginning of the conversation to map out All their various concerns and now the developer knows what they need to do to do and the rest of the folks Have had their input So the application that gets produced at the end already has the necessary controls in place It already has the right infrastructure that it needs rather than trying to slap all that stuff on at the end when it's much more difficult because Some of the decisions made early on in the process would have to be reversed or drastically altered.

So I think That's a big component of it is bringing folks in earlier on in the process and then embracing an automation Workflow for the actual development of the application and the deployment of the application That's using all the principles and information that you learned at the beginning of the exercise So I think if you adopt that sort of mindset, you can have a really good outcome But if you say Oh devs, you're just responsible for all this stuff now And why didn't you get this weird esoteric compliance thing, right?

You're just gonna burn out your devs and they're gonna go find a job that plays just as well and isn't asking them to do So going back to our original in closing out going back to our original Use case which was the Portworx replication. I don't like the concept of technology. That's that what we're basically hearing from you Ned is that the Replication and the ability to do this capability is not bad. It is a good thing But it has to be something that's Intentional it can't be something where the developer of the application comes says.

Oh I can bypass all of my existing IT shackles implement replication and get my cluster highly available without being slowed down by IT sometimes it's okay to be slowed down because if you can't if you don't have this skill within your Level Responsibility to support it when it breaks It's best to engage in the people who are experts in this from a practical sense whenever I look at a new replication technology Inside of my data centers I would going in engage the networking folks and I'd engage the storage folks So one I make sure there's enough network to do the replication That's probably a good thing to know and to see if there's any other paths so should I use the traditional namespace DNS namespace to replicate or if I use a Alternative namespace is a is there a dedicated set of circuits or links to do replication?

So I can ultimately meet my SL links So that we're at time we try and keep this to about 20 minutes or so and we're at that we're at that point and if people want to learn more about developing and developing DevOps practices Or cloud governance kind of day to where should they where should they look at? Which is very subtle These are a lot of the topics that we try to cover on the day to cloud podcast so you can find that at Day to cloud IO.

That's easy to find I also have my daily thoughts on my daily check-in Which you can find on anchor FM just search for a daily check-in with net 1313 And if you want to actually do a little education learn a little bit about maybe terraform or vault or Microsoft Azure You can check out my courses on Pluralsight All right, if you want to find out more about the CTO advisor You can find us on the web the CTO advisor comm you can find me on Twitter at CTO advisor DMS are open you can find net on Twitter at net 1313 Really easy Twitter having to remember until then talk to you next CTO advisor podcast You