Configuration Management for Hybrid IT environments
Transcript
>> Hello, and welcome to this session, on Configuration Management for Hybrid IT environments. My name is Martez Reed. I am currently a Principal Training Solutions Engineer, at Puppet. Focused on internal at external enablement for our products. I can be found on LinkedIn, Twitter, as well as GitHub. Before we dive into configuration management, let's talk about business outcomes. Every IT organization should be focused on helping the business reach certain outcomes. And this may be ensuring that the products, or services that the businesses provides, deliver a certain value to their customers.
In addition to that, ensuring that the products, or services are available when a customer expects them. As well as also providing a level of trust to the customers, and this is especially important when talking about, IT security and data privacy. Every business has some sort of, way for the customer to reach them, to purchase products or services, often is a website. So in this example we'll start to walk through a website, and how configuration management plays a role in delivering value to the customer.
The Application Architecture might be an Nginx web server, that fronts a Python application, that ultimately talks to an Oracle database. So we'll break down this architecture, and see where configuration management is used. So the operating system needs configuration as well as hardening, and so assuming that each of these lives on three separate virtual machines, there's work that needs to be done there. Also monitoring, logging, security agents, need to be installed on the machine, as well as the actual application configuration and installation, so the case of Nginx, ensuring that the Nginx package is installed, it has the appropriate configuration for the web server, as well as potentially moving, some static web files to the server itself.
Case of the Python application making sure, certain library are installed, as well as the actual application itself. And then only Oracle database, making sure Oracle is installed, as well as having the appropriate configuration on system. So this is where configuration management tools, can come into play to help configure all of those pieces in a codified manner. So in this space, you have Red Hat Ansible, Puppet, Chef, as well as Saltstack. There're a number of other configuration management tools, but these are usually the main four that are talked about, when talking about configuration management tools.
And the goal of configuration management tools, are ready to help define, the desired state, of a machine in a codified manner. So, there're different architectures, for how this is done. So there is the agent based architecture, or the pull based model which utilizes, an agent installed on the machine itself, that checks-in with the master, or a server, or centralized server for its configuration, or desired state. And then the server returns the desired state configuration, at which point the agent on the node, works to reconfigure potentially the, all the settings defined in the configuration.
In addition to this, there's the agentless model, which Red Hat Ansible utilizes, or the pushed base model. So what this is, is that there is a control node, or a machine that has Red Hat Ansible installed on it, with a desired configuration, at which point it then pushes out the desired configuration, to the node, and interacts with the node to configure it to the desired state. So in the case of our application, we would be using our configuration management tools, to configure the application, in the middleware that needs to be installed, as well as configuring the operating system.
So, in our own approved environment, this is where the configuration management plays a big role, in configuring the systems once they've been provisioned. So let's talk about provisioning. Traditionally an on-prem environments we're utilizing, a cloud management platform, for automation purposes. So cloud management platform, helps us to provision things, in an automated fashion as well as integrate, with our configuration management, to provide an end to end solution for our automation needs. So some examples are vRealize Automation, as well as Morpheus data.
These applications provide portals, to request services in an automated fashion that align with what the business needs, for developing new applications. So an example of provisioning workflow, that we might utilize on-prem, might be vRealize Automation talking to VMware vsphere, to provision a virtual machine. Then it reaches out to NSX, to add the appropriate security groups, from a tagging prospective, to ensure that our application is secure from a network communication aspect. And then we might trigger a Puppet Agent Run, to configure the actual operating system, level configuration, as well as the application and the middleware pieces.
And then following that we might utilize service now, to create a CMDB record, or the integration with service now. And so, this workflow should help us reach our business outcome of being able to stand up and access the website once that full end to end process has been completed. And so the role that configuration management played was, taking the hand off after the virtual machine was provisioned, and then configuring the instance to finish that final goal, of providing a web portal for our customers.
So now that we walked through the on-prem example, let's take a look at how things start to change in the public cloud. So for a lot of businesses, there're new business outcomes, that the business is trying to reach. So this maybe retaining existing customers, introducing new services to customers, and then expanding into additional industries or markets. In this example, we look into expand, beyond selling TVs, into now selling food. And so every business or organization, begins to have a cloud journey, those that have been to the public cloud.
And so let's talk about this. So as an example we might be looking to leverage, PaaS offerings to speed development and reduce administration. Reach new global markets through regional proximity, and then quickly scale to meet increased demand. " So in this example, we're going to go with, assume that the business went with AWS. And so now we've got to connect the two clouds, in some sort of manner. For many organizations it's a direct connect, for network connectivity between the two, or a VPN to connect the two.
But let's just assume it might be either one or both, and now that we've connected the clouds, we sort of start this process of working on developing our hybrid cloud. " So in this case we've got some different options, for provisioning work loads in the cloud. Similar to on-prem environment, we've got CMPs, working for cloud management platforms, which might be a Servicenow orchestrator, Morpheus, or again vRealize Automation. To extend our automation, or our provisioning automation, out into the cloud to provide the hybrid architecture between the two, and provide that consistency.
We may also go the route of looking to utilize a CLI, or a program language like Python, PowerShell or GO. To actually provision the instances where the hardware, or things in the actual cloud itself. In addition to that, we might look at infrastructure OS code as an option, so Terraform CloudFormation Pulumi, aren't templates, those things to provision our infrastructure. So this is where we start to have, different decision points as we make the cloud journey. So for the new world, I may decide that instead of leveraging the existing cloud measurement platform, whether it's cost, whether it's the desire to utilize an infrastructure as code language or paradigm, or whether it's just a decision point between different teams to move away from potentially what is leveraged on-prem.
So our next example, we'll assume that HashiCorp Terraform is utilized. The decision was made because of it's multi-cloud support, its large community and numerous code examples, as well as its fast support for new vendor services. So infrastructure as code provides, a codified manner in which to define, the actual resources in the cloud. So I can define my VPCs, my actual EC2 instances, my Rds instances in a codified manner. Now that we've determined how we're going to provision our infrastructure in AWS, let's take a look at our applications.
One option is to potentially utilize Lift and Shift, which is the thought or process of simply moving the existing application architecture to our public cloud environment, and basically mapping our virtual machines, on-prem, one to one to EC2 instances in ADVS. In this case configuration management would help, simplify this process and standardize it, the configuration, across both environments. This is where a lot of companies start to look at, how they're consuming services, especially for the public cloud, and what it has to offer. This is where we talk about Consumption Models.
In the Lift and Shift model it would be IaaS, which is Infrastructure as a Service, or you would be leveraging virtual machines, or EC2 instances. In addition to that, cloud providers begin to write additional value by offering things like, PaaS as well as SaaS. PaaS would be Platform as a Service and so it would be things such as manage databases or manage elastic cash services or manage big data. So this helps reduce the administration overhead, as well as speeds up the process of initial configuration to actually harness these services.
In addition to that, there's SaaS, or Software as a Service, which is sort of a complete solution where you're simply just accessing a providers application and some examples might be Workday or Salesforce. In this case now the business would like to begin taking advantage of a PaaS offering provided by AWS, in this case it's Amazon RDS, to provide them with a fully managed database offering to eliminate the need for them to handle a lot of the day to day administration of their database, as well as being able to offload that process to AWS.
In addition to that we would still have Puppet configuring our Nginx instances for our web servers, as well as our Python application. And so this is where we start to move from the standard Lift and Shift, to beginning to take advantage of native offerings provided by the public cloud. And so in this case Puppet would be handling the configuration of our web server and our application server, but we would be utilizing HashiCorp Terraform to actually provision and configure, essentially by providing standard settings for our AWS Amazon RDS instance.
And this is where it starts to change from our on-prem environment where Puppet was being utilized, or our configuration management tool was being utilized, to configure all of our instances, or our virtual machines, at least at the operating system level and application level. And so now in this case we're starting to shift from that. And so another thing that typically comes up when starting to move to the public cloud is Immutability. And what, immutability is, or immutable infrastructure as we begin to talk about it, is really a management paradigm that focuses on the recreation of an instance to push new configuration, or applications, versions, or to actually resolve an issue.
So instead of actually fixing what might be, the underlying problem with the actual instance itself, we would just create a new instance and essentially throw away the old instance. So this is ideal for stateless applications, or applications that offload their state to something like an AWS S3 bucket, to eliminate the need for the state to be very tightly coupled, to the actual life cycle of the instance itself and this is commonly referred to as, treating instances as cattle versus pets. And now we're going to look to leverage this paradigm for our web server.
And now we need to bake everything, or install everything, into our Amazon Machine Image, or our AMI, prior to actual deployment of the AMI itself. And so this is where we would look to utilize a tool, like HashiCorp Packer to codify the creation of our AMI and pre-install all the software that's needed, so that way when our web server gets spun up or provisioned, it then has everything it needs to start servicing requests. So it might choose HashiCorp Packer for it's multi cloud support as well as it's large community and numerous code examples similar to HashiCorp Terraform.
And so how this would work might be, utilizing HashiCorp packer to provision an instance, and then potentially triggering an Ansible Playbook. So this is often where we start to begin looking at incorporating new tools. So in this case we might look to leverage Red Hat Ansible, because of it's push based model, for pushing configuration to instances. In this case the instance is going to be Immutable, in the sense that I have no need for on going management of it so I may look to leverage a solution, or a tool, with a push based architecture.
In addition to this, the existing pull based model of a Puppet or a Chef, could still be leveraged to pull the configuration but would require the removal of the agent, or utilizing similar mechanisms where the agent is installed but does not leverage a master and utilizes a master list, or in the case of Chef a zero, deployment of the system configuration. And so what this is going to do, is now allow us to still leverage a configuration management tool, for our Immutable infrastructure. At least the configuration or deployment, or installation process at the software and everything that we need.
So in this case we'll look again, at how things have shifted from our on-prem version of the application architecture. So in this case we're leveraging HashiCorp Terraform, to actually perform the configuration of our Amazon RDS instance. Still utilizing Puppet to install and configure things on our application server. And then now looking to utilize Immutable infrastructure with HashiCorp Packer and Red Hat Ansible, to perform the configuration on our web servers, or Nginx servers. So now that we've looked at that, let's take a look at secrets management, and how it changes when we introduce public cloud.
Half the times in the on-prem environment, we might have been leveraging the native configuration management, secret management within the tools itself. So in the case of Ansible, there's Ansible-vault, in the case of Puppet, there's Hiera-eyaml, as one of the notable secrets management tools. And in the case of Chef, there's Chef data bags, which could be encrypted. And so all these encryption mechanisms, or secrets management mechanisms, are native to the tools themselves. But one of the things that we still have to look at, or rationalize as we move to the public cloud, is how this might change with new cloud native services, that provides secrets management, as well as looking to unify our secrets management strategy across the environments.
So this is where we begin to talk about, products such as HashiCorp Vault. For secrets management and data encryption. Cyberark Conjur, and then potentially look at, cloud native or native secrets management provided by, the cloud provider themselves, in Azure Key Vault, or even AWS secrets manager. And so, one of the things we have to rationalize is, one, the integration, with the configuration management tool. And some examples it might be like a HashiCorp Vault, which is well integrated with a lot of the, configuration management tools, like the Ansibles, the Chefs, the Puppet, to provide that secrets management across, both the on-prem environment, as well as in a cloud environment.
So you have to rationalize whether it's continuing to utilize multiple desperate secrets managements solutions or often times looking to unify that secrets management across environments. And so this would require additional work, if there was a desire to unify secrets management across environments. As I said, the integrations are there for HashCorp Vaults, and then there's integration for Conjur. And then even utilizing some of the public cloud, provided secrets management solutions, with the configuration management tools. But it is certainly a topic of discussion that needs to take place.
In addition to secrets management, often times when organizations are going to the public cloud, there is often times a rationalization, that is made around selection of an operating system. Often times there may be, existing operating systems that are leveraged. And so as you begin moving to the public cloud, you look to make decisions around, what you're going to do from an OS perspective. And so some of the decisions might be centered around, reducing licensing cost, standardizing potentially on a single OS platform.
And then leveraging existing expertise, and so in certain scenarios it might be, I have a existing Red Hat enterprise linux work loads, that may be moved to CentOS or CentOS operating system, or potentially moved to an Ubuntu, or whatever flavor of operating system may be desired. But this does certainly have ramifications, as you talk about configuration management. The ramifications of migrating to a different OS, should hopefully be pretty minimal, especially with configuration management, helping that process. Most configuration management tools, are written in such a manner, where there's an obstruction between the actual underlying operating system, and how we declare or define our particular resources.
So in the case of a package, I might say package install Inginx from a process point. And then that maps to APT, or an Ubuntu, or it maps to Yum on Red Hat, or CentOS. Some of those servers, I might be managing the Inginx servers. And in this case it might depend upon the actual playbook, where the module, or the cookbook to map that to, a tiger operating system. This is where there is some new antix there, if the particular configuration management artifact, or the cookbook, or the playbook, or the module, isn't bringing the manner to easily support, most operating systems, there will certainly be the need to be some work done there, to support both platforms or multiple platforms, if the decision is made to move, to a different operating system.
Let's look at application selection. And by application selection, this is usually the rationalization of various tools, or platforms within an organization as they move to the public cloud. So an example might be, currently Splunk being utilized on-prem, and maybe the decision is made to utilize Amazon Cloudwatch, in the public cloud environment. This might involve the actual installation of an Amazon Cloudwatch agent on the instances. And so this would mean, potentially trading consent, or looking to integrate new consent from a configuration management perspective to install and configure that agent.
Similar in the case with a monitoring, with existing Sensu, and then moving to DataDog. In the case of maybe vulnerability management, it might be a case of utilizing a Qualys agent, on virtues machine and on-prem environment, and maybe instead of actually installing an agent in the public cloud, maybe its looking to leverage, a golden master, or a golden template, for scanning purposes. Scan that, and then after that has been verified as good, then moving away from actually installing an agent on cloud instances.
Maybe it's the same with Centrify, for access management, or maybe it's utilizing jump host, as the single point of entry. And that's where Centrify is installed, for accessing our EC2 instances. And in the case of our Inginx servers, where we've gone to enable a paradigm, maybe there is no access to those. Maybe it's a case of, once it's deployed, then it's either killed off, or it just remains as is. And there's actually no one touching that, so there is no need for access management, directly to that instance itself.
And so as organizations journey to the public cloud, there is often a rationalization of what tools, and what agents, and what software is going to be installed. And this has an impact on configuration management, from the perspective of what applications are going to get installed and what code maybe, either need to be tweaked or written, to actually support that new model. Let's talk about artifact storage. And we're talking about artifact, usually we're either referring to a third party software installation package, something like Microsoft sequeled installer, or potentially an artifact of a software development process for an internal application.
We're talking about artifact storage, generally this is when the conversation, takes place to artifact repositories. Artifact repositories are repositories or locations for storing software packages or artifacts. Some examples are JFrog artifactory, Nexus repository, or ProGet. And where this relates to configuration management, is generally the configuration management tool, will direct the system to pull the artifact or the software installer from an artifact repository. In addition to this, some examples, are things that are used or also NFS shares, or Chef shares for storing the installation packages.
When we start to talk about moving to the public cloud, we have to begin looking at different options for actually where we store these artifacts. Lot of cases, it's in Blob storage or object storage. Such as an AWS S3, Microsoft Azure Blob Storage, or GCP Cloud Storage. The advantage of these is that they're managed by the public cloud providers, that they're integrated with the existing security access models, as well as infinitely scalable. " And where this starts to impact configuration management, is that the code that's written, is likely referencing in the case of, our on-prem environment, it might be JFrog artifactory platform.
But in the case of AWS, we might look to leverage AWS S3. And so now we need to write new code to reference that. In addition to this, we might also look to leverage potential integrations, that are provided by our repository manager to reference a location or a distribution point in the public cloud to help reduce some of the additional code that needs to be created. T. space. So Containers are essentially, portable application environments. So the goal is to package up the application as well as it's dependencies in sort of a single artifact.
" Containers help to eliminate this by bundling everything that's necessary to run the application in a single package. Usually, when we're talking about Containers, we're talking about Docker Containers. There're various other Container run times that can be used to run Containers. In addition to Containers, we have to start talking about Dockerfiles. And this is where we have the sort of conflict between Containerization and traditional configuration management. And so in this example there's a Dockerfile, and it basically references or specifies everything that's needed for the actual application to run.
And so after this I might run a Docker run, after I've run a Docker build to create the container image, and then I'll run a Docker run to actually run the Container. And this process involve no traditional configuration management, and so this is where we have to start to look at, "How can Configuration Management and Docker play together? " In addition to Docker, we also start to look at things like Kubernetes for our Container orchestration to run Containers across multiple nodes and provide the redundancy and resiliency that we would expect of our applications.
mo files that define how our environment is going to look, or application is going to look from a code perspective. So in the case of Kubernetes, I may decide Configuration Management doesn't have it's traditional role, and maybe I leverage it for rolling out Kubernetes environments, of the Kubernetes clusters, or different integration points between both Configuration Management as well as Kubernetes. It may be the point of deciding that Kubernetes is going to be leveraged outside of what I'm doing in Configuration management. This is where we start to talk about Microservices of compossible units of applications being our go to model for our application architecture, as opposed to a monolithic architecture.
" This where we start to talk about how configuration management can help as part of a hybrid environment, that includes microservices, containers, Kubernetes. So there're scenario where a virtual machine needs to talk to a container or a service in the public cloud, or it's also hosting a Kubernetes environment in the public cloud, then we look to potentially leverage something like, HashiCorp Consul, for service discovery, so that our virtual machines can find the service endpoint of our particular microservice application. Because the containers that underpin the service itself, are likely very ephemeral in the sense that, they can die and be recreated at any time, so there's no static sort of IP address, or DNS name that's often associated with it, so I'll leverage HashiCorp Consul service discovery to be able to find that service wherever it might be in my Kubernetes cluster.
In addition to that, I might also use HashiCorp Vault to actually access maybe the API endpoint from a secrets or credentials perspective. And both of these include agents that can be installed on my virtual machine via my configuration management tool to actually help facilitate this process, of sort of blending both the old world and the new world. So this is what an example might look like, where I leverage Puppet to install HashiCorp Consul, or Istio agents on the virtual machines to be able to interact with my containers that have been configured utilizing Helm charts.
And so I'd like to think of this as sort of the blended sort of scenario, where most enterprise organizations will need to be at, where you need on-prem and sort of public cloud, native container, Kubernetes, to be able to work together in a homogenous fashion to be able to harness the capabilities of both environments. Thank you for joining my session. (dramatic tune)
