Scaling Infrastructure as Code to Improve Delivery & Maintain Security
Transcript
>> Hi, my name is Rosemary Wang, and today I'm going to talk about Scaling Infrastructure as Code to improve delivery and maintain security. Seems like a pretty generic title. But it really started because I walked into an architecture meeting, I saw this big red line on the board, and I saw some boxes representative of applications and databases. " They said, "Well, anything above the line belongs to the public cloud infrastructure that we've deployed. " And I think that what they were trying to get at was that it was different because it's a different way of engaging with infrastructure.
In hybrid IT, you've got this interesting mix of systems of engagement. Ways that you use offerings, ways that you leverage on demand, very dynamic infrastructure. And then you've got systems of record, right? In the private data center, you keep track of your inventory. Because you've procured it. You've dedicated resources to it. You've ensured that you have enough resources to support the applications you need. So you really need those systems of record. So when you're in hybrid IT, you sit somewhere in the middle between a system of record, and a system of engagement.
Which means well, if these are technical systems, your organizational systems are also going to engage very differently with these two infrastructure footprints. " You get different ways you interact with Public Cloud Infrastructure, and different ways you interact with Private Cloud Infrastructure. " How do I make this infrastructure at the end of the day? So in that journey, I moved a lot of people and ran a lot of systems in hybrid environments. And I eventually became a developer advocate. I thought that maybe if I became a developer advocate, I would be able to have a cure all to this problem.
But the reality is that it's very nuanced, right? Most people start with thinking about Public Cloud Infrastructure, and they say, "Let me try Infrastructure as Code for it. " But the reality is that learning 'Infrastructure as Code' and then applying it right into your organization is a little haphazard usually to start. You don't always have the best practices and the best patterns in place. And when you grow it, it becomes an organic growth. So then when you try to backpedal, you take this Infrastructure as Code idea and you try to push it onto anything below the line, which is anything in their private data center.
Well, it kind of breaks down. You stop thinking about the principles that are central to Infrastructure as Code and the software development principles that founded it in the first place. So there became this question, and this developer advocate that I tried and I still try to answer, which is how do you scale Infrastructure as Code? In the case of hybrid IT, scaling is a big issue because you've got more infrastructure with a greater variety of how it's managed. So really, how do you actually scale it?
Today, what I'm going to do is actually answer sort of the three outcomes that you want to get from scaling your Infrastructure as Code. Because by taking away the line by saying, "I've scaled my Infrastructure as Code" the hope is that you're able to support the (murmurs) and drive continued business value as your organization continues to grow. So let's actually think about the three outcomes. When you adopt Infrastructure as Code that you're looking to do and we'll apply it back to some of those core software development practices that we had previously mentioned.
So there are three outcomes that I see for when you apply Infrastructure as Code to your organization. " By developing consistently, it means that you're making changes to production with low-risk and you're deploying securely. So these are the three outcomes that you're hoping to achieve with the idea that it will drive additional business value and growth later on. So you need to be able to scale all of these three outcomes. And I'm going to actually pick some practices that are really specific to driving some of these outcomes and explain a little bit of why and how you scale your Infrastructure as Code to accommodate and build out these outcomes.
" Delivering quickly is in the realm of continuous integration delivery or deployment and automated testing. The idea is that I need to be able to deliver quickly and make sure I manage the risks associated with it. So that's the second piece, which is testing. Now in your organization, you're going to have a number of teams with varying continuous journeys. So you can choose your continuous journey. Some teams will find themselves in continuous integration. The idea is that these changes will sort of be held in isolation for quite some time.
They'll get an approval and then you'll take all of those changes and push it to production. That's continuous integration. In continuous delivery, you actually have one change that's maybe reviewed. It's good to go and it's pushed. So maybe it's on a schedule, but on the other hand it's pushed a little bit more frequently. So you'll hold some of these changes and then you'll push it to production on a regular scheduled cadence, which is pretty typical. " Continuous deployment's a little bit trickier and a lot more difficult to adopt within an organization that's focused on risk mitigation and infrastructure.
It's also something that it's not going to be possible for all infrastructure. " So in the case of Terraform, which is the HashiCorp tool, you would do a plan, you would not check the diff at all, you would trust that the diff is fine, the differences are fine, and then you just apply it straight to production. So there's no manual approval at all. There's no change review board, there's no change review at all. But it doesn't always work in every piece of infrastructure.
Forget about whether or not it's data center or in Public Cloud. The idea is that some changes, some types of infrastructure will not lend well to continuous deployment. And in that case you will probably not have teams in your organization who will do continuous deployment at all. So how do you assess what infrastructure fits on which end of the scale? As I mentioned before, things with smaller blast radius, well-defined rollback, a very high rate of change lend really well to continuous deployment.
It means that you don't have anybody necessarily clicking off the manual approval, you're always going to deliver it. But then you kind of like exist usually somewhere in between in your infrastructure organization as you scale Infrastructure as Code, you'll tend to maybe deliver on a schedule. You'll tend to look and aggregate some changes and just push them out as frequently as you can. But then there are some pieces of infrastructure. This is really critical infrastructure for example, that you do not want to just push in and you know, automatically.
You want to make sure you do due diligence and understanding the change, what kind of blast radius it has. If it has a particularly challenging rollback, for example, if you have BGP routing changes, that's not something that's easily, continuously deployed or even continuously delivered. So the crux of this right, no matter which of these patterns you choose, you have to do automated testing. Infrastructure testing is extremely difficult. I think that there are a number of people who have tried to articulate the ways that you could approach this and it's not completely intuitive.
So this is a testing pyramid. It is a basically a indicate, it's a helpful indicator and framework for the way you think about your testing strategy in software development. In software architecture, you often discuss a software testing strategy. So do you commit to Unit tests? Do you commit to End to End tests? In the case of infrastructure, it's very difficult to commit to End to End test for everything. It's very expensive. I remember that there's only been once that I've seen that there's a full network mock network, staging network created just to mimic changes to production.
It's very expensive. There's a time and effort to maintaining as full staging environment. So the result is that you actually in infrastructure have a testing strategy that has to be a little bit more focused. And I usually focus on integration and contract tests as part of testing strategy for infrastructure. The reason why is that integration tests are a great way to understand what's happening on the line to use that term, it's a great way to understand whether or not you've broken something as part of a change between that border in your Public Cloud Infrastructure and your Private Cloud Infrastructure.
So making investments into those kinds of tests, very selective, very specific to isolation. It's not End-to-End. It's very specific to infrastructure components on that border that will actually help expedite your testing and lower your testing cost. On top of that, I usually lean on contract tests and most Infrastructures as Code usually leverages some kind of state system. So the use a state system to basically capture information and metadata that becomes a source of truth for the current state of your infrastructure system.
Contract tests are quite great way to understand whether or not you've affected the inputs and outputs that go into state. So more importantly, let's say I've changed a database, well, does that change mean I'm updating the database or does it mean that I'm actually destroying it and recreating it? In which case I've broken a contract where I'm not supposed to actually take down that database. So it's important to invest into these kinds of tests as part of your overall infrastructure organization's testing strategy.
Now, besides delivering really quickly, you also want to develop consistently. This is because that if someone is making changes and they're doing it in a different way in your infrastructure, that often leads to conflicts. In the case of developing consistently, I usually refer to two things. Development models and code or dependency management. Development models are very difficult to articulate in the software space, but it's also in some ways even more nuanced in the infrastructure space. So in the software space, when you talk about feature branching, usually you're branching per features.
If you want to develop something new, you create a new branch, you'd build on that in isolation. You run some tests and then you win the test pass, you merge them back into the master, the production kind of production development code. But in the case of infrastructure, there's actually two ways of feature branching. You stage per branch. So that means that on demand infrastructure is created per branch. In this case, this might be Kubernetes clusters that are created at will for every single branch that you have or you have staging infrastructure dedicated to branches.
So if it's something not master, it will always point to dev network, staging network, something like that. The reason why you would defer to one or the other is that you would maybe want to say that some infrastructure you cannot get on demand. The other way that you could approach this is this is also dedicated infrastructure already exists. So this is something that you may think about in the case of more static infrastructure. In the case of trunk based store, trunk base actually means that you go straight to main line.
So there's only one code, a branch which is just master. And in this case you deployed a staging before production. I have to understand whether or not that code path is a problematic code path. So let's say I made a change to a network and it is an invalid CIDR block, I would be able to catch that in theory and staging before it ever got to production. So that pipeline fails immediately. In this other cases when some people don't want to have a staging infrastructure environment due to cost, they deploy straight to production with the expectation that there's an immediate rollback on failure.
So why bother going with one kind of development model or the other? In your organization, you're going to have teams that do both. You're going to have teams that do feature branching and you're going to have teams that do trunk-based. And then the reason why is not because the teams are of different practice. It's usually because there's different types of infrastructure for them to use. So in the trade case of Trunk Based Development, it works fairly well with very dynamic infrastructure.
This is not just public cloud offerings, it's also for let's say Kubernetes resources as well. And part of this is that you really want to focus on trunk-based for immutable infrastructure, infrastructure that has a very high rate of change, and has really strong automated testing around it. Trunk-Based Development has a deep focus around ensuring you gain confidence through automated testing. However, there's the other part of this where Feature Branching, while it seems good in theory, the tricky part about the Feature Branching is that you have to know get really well.
In the Infrastructure as Code, you're usually depending on some form of version control and in that case you have to have a deep understanding of whether or not someone's conflicts will affect yours. Whether or not the changes that someone has made will impact yours. If you are using static infrastructure, these changes may not go out for three or four weeks. One good example of this is that, let's say I was making a network change that may not go out for three weeks and in which case I have three weeks of someone else making changes to that environment and I would not necessarily know if it affects my change or if that conflict could be properly resolved in production.
So development models are actually really crucial and it takes a very careful assessment of whether or not certain infrastructure is worth going one way or the other. Sometimes you might find yourself actually doing both patterns in the same exact team. The next piece of this that I think is important to understand is Code Management. When you start using Infrastructure as Code, it's easy to just put everything in one repository because that's everything you see. It's very easy to learn, but it actually grows out of control.
So known problem. And people will tell me, I have 200 Terraform files and I'm like, "Oh, that's not helpful. " When you actually look at your giant body of infrastructure configuration, identify a bounded context, take a domain driven design approach to it where you can identify the lines by which you should modularize. First bit of this is blast radius. How wide is this change If I make it? If I maybe make a change to a network that's actually something that's very wide versus something that's very localized, like a serverless function, the rate of change is also really important as well, right?
So just as much as I don't make that many network changes, for example, I may make a lot of serverless change, serverless function changes, in which case I would want to separate those two code configurations. The other bit is scope of responsibility. So if you have, let's say Kubernetes are back and AWS or GCP or active directory roles and responsibilities and access control, those would have to be separated. So you want to make sure that you're dividing along those lines. Also, ease of management is extremely important.
If one set of infrastructure is easier to manage than the others, that's a clear line to delineate, right? So if you've got an Oracle database that's in the cloud, it's a little bit easier to manage than maybe what you do is divide that into a separate code body versus if you're doing this in an Oracle database in your private data center. Finally, how in the world did you actually share all of this standard configuration? I think it's important to realize that when you go along this evolution of dividing out the code for your infrastructure correctly along certain lines, you realize other people want to use it too.
This is really common for let's say database, lets say acutes where you're going to have multiple people asking, "Can I have a database? " And in which case you maybe want specific standards from an architectural standpoint, you want to maybe ensure that it's always encrypted, that it's always private and it doesn't matter if it's in data center or a public cloud. The idea is that you want to make sure that these are golden standards for the way you manage and configure infrastructure. So which case you offer them as sort of their own modules as dependencies.
And in that case, you start to build this concept of versioning the configurations, making sure that you actually say, "I made an update to their fry version, it's a new module" and you have to maintain schemas for inputs and outputs. A good example of this was that, I made a change to a vault helm deployment on GCP for example. And someone who decided that they were using this to basically help their on premise testing, they said, "Oh, well I don't really want to upgrade to the newest version of your module yet because it has a new input that requires.
" So it's important to version and it's important to maintain schemas for inputs and outputs. The idea is that this enables someone to do self service so someone can actually take this module, create it themselves. And it also creates a common architecture. Someone's not going to have to sit there and duplicate this later. Someone at least understands the architecture that you expect from this. This is a bit of a divergence, but I do want to actually bring this up as part of developing consistently.
There is a big difference between Mono and Multi repository. In the case of Terraform for example, a lot of people start by putting 200 Terraform files in one directory and eventually they'll apply a bunch of Jenkins change sets to basically recursively search through all these modules and apply changes. It's at some point or another and not terribly efficient. If you have a good Monorepo Build System and this kind of model works very well so you can at least isolate and construct and selectively organize your code modules for infrastructure locally.
And this is pretty effective. However, most people don't have great Monorepo Build Systems. So instead, Multi-repository is one way that you could go and the idea is that you would store the binaries for these modules either in Version Control, in Artifactory, in the let's say, Terraform registry. They're versions you can reference them remotely, they are retrieved. Sometimes if you're using 300 of these modules, yes, it will become a little bit unruly to grab them from the remote end points. But I do think actually if you can still, for the most part compose these modules correctly such that you don't actually have to pull 300 of them.
Finally, deploying securely. This encapsulates Security as Code and Policy as Code. And I always bring this up because someone told me we should be begging security and we shouldn't be just, attacking it on at the end of the day. And I always borrow this jazz, "Humble Matrix" of Sotts where he kind of did the three quadrants, which is 'Known knowns' are testing, 'Known unknowns' are monitoring, and 'Unknown unknowns' are observability. So 'Known knowns' would be something like, "I know that my server should be of this image of this CIS compliant image" that you would test.
'Known unknowns' would be, "I need to make sure that this server doesn't run out of memory" so that you would monitor. The 'Unknown unknowns' would be, "I want to know that this transaction actually reached this server" because that's on demand, I need to know that disability right then and there. The problem is that organizationally we actually have something called, 'Unknown knowns' which is siloed knowledge. It's knowledge that's in the hands of a few people. And not to say that they don't want to give the knowledge or not want to disseminate the knowledge, it's just not something that everybody knows very easily.
And how many people actually go into a Wiki page and read on all of the Security Standards that they need to uphold. It's very unlikely. As a result, as your organization grows, as you scale your Infrastructure as Code, these kinds of 'Unknown knowns' they don't scale anymore. So there's this idea of evolutionary architecture. The idea is that your architecture should support the evolution of your system. So if you prioritize certain capabilities or certain architectural priorities, your evolution of your architecture should reflect that.
And I think in this case, it's really a great way to frame some of the Security as Code and Policy as Code approaches. Policy and Security are Unknown knowns. They're things that are siloed knowledge, their architectural capabilities and architectural priorities that aren't always clearly articulated. So how do you articulate them and scale your Infrastructure as Code? Well, you do them as code as well because they are as much as part of your infrastructure. So in this case, I show a Sentinel rule.
Sentinel is available on Terraform cloud which is the SAS offering for Terraform. " It means that if there is a premise OF Firewall rule, then I should not be pushing this to production, then this is not a valid change. Similarly, let's say from my policy team, I want to make sure that everything is tagged, all the resources are tagged correctly. And the reason why I might do this is for chargeback, it might be for audit, it might be for just making sure that there's no bad actors in the system, but regardless, this is all in a way to encapsulate security and policy knowledge in a codified way.
And the reason why you do this in a codified way is that it does scale. So in the case of Terraform, for example, what you can do, is you can run a plan. So a plan just dry runs, whether or not it will make a change, it will show you what changes. But normally will you just say, "Then we make these changes" you can apply these Sentinel policies to say, "Is it a change that's valid? " And it's really powerful to do, right?
Because if let's say, you are configuring a PCI compliant network within your data center, private data center, you would want to know if let's say some rule or some information or some kind of configuration would no longer be compliant with PCI. So in that particular situation, this is a way that you'd be able to apply it depending on the type of infrastructure you have, given that you've modularized it. And also the configuration at no matter where it is, it could be above the line, below the line.
The idea is that these policies are able to capture the information that is usually siloed and allows you to scale your infrastructure to support all of the things that your organization wants to do. So at the end of the day Infrastructure as Code, you implement this in your organization because you want to enable the self service engagement with the system. You want to move away from maybe the system of record. " The reason why you say self-service is that as much as we say that our organizations inform the way that we built our systems, and that goes the same way with infrastructure systems, infrastructure systems also seem to inform the way we built our organizations.
And part of this important scaling discussion is to actually normalize the patterns across the organization. It is not to say that you must become continuous deployment of infrastructure. It's not to say you must do all trunk-based development. The idea is that you can identify some of the patterns within different teams, within different types of infrastructure. Understand how it fits in, how it really might help your risk management within that kind of infrastructure component. Understand the limitations, right? And by normalizing some of those patterns, you can actually scale your Infrastructure as Code more effectively.
You can identify even which private data center components below the line are kind of giving up the holdup, are stopping you from delivering quickly. And you can make improvements to that. So I think at the end of the day when coming back to normalizing patterns and identifying within that architecture team, what was really below the line and above the line, we had started to understand the first stage of scaling our Infrastructure as Code so that we actually had a holistic experience and we could build this self service system of engagement.
Instead of just saying that, we were always going to be hitting this line and halting before we hit a certain part of their infrastructure. So if you are curious to learn more, I have plenty of resources here, and there are some technical deep dives on this page as well. If you're curious, you can dig into it and learn a little bit more. But I hope this was a general overview and gave you some interesting thoughts on how to approach Scaling Infrastructure as Code in your organization.
Thanks. (metallic music)
