I Broke The Data Center – DNS
For the past week, we’ve had this pesky routing problem in the CTO Advisor Hybrid Infrastructure (CTOAHI) data center. We finally had some time to take a look at it. We needed to make a disruptive network change to try to fix it. We’ve prepped the environment for the past few weeks as we’ve migrated all of the storage to our shiny new HPE Alletra storage array. Finally, we felt confident, and it was at this point it all went wrong.
Before the HPE array, we ran all our virtual machines (VMs) on VMware vSAN. Without knowing a lot about vSAN, just know that losing all network connectivity is bad. So, our mitigation is to have it on a proper storage array.
So, we went ahead with the network change. All of the VMs lost connectivity to the storage array. That was an expected risk. What wasn’t expected? The DNS ran on a VM that was on the storage array. Our playbook has a process where we connect to the vSphere host running the DNS VM and manually boot it.
We didn’t realize that the new storage created a new dependency on DNS. The vSphere hosts required DNS resolution to connect to the storage array to boot the DNS VM… A Veeam restore later, I had the DNS VM back up, but the damage was done.
We are deploying a physical DNS server as redundancy to avoid this problem in the future. However, the hard lesson is reevaluating the dependencies when migrating critical systems. We might have avoided this if we had done a tabletop exercise.
Share This Story, Choose Your Platform!
Keith Townsend is a seasoned technology leader and Founder of The Advisor Bench, specializing in IT infrastructure, cloud technologies, and AI. With expertise spanning cloud, virtualization, networking, and storage, Keith has been a trusted partner in transforming IT operations across industries, including pharmaceuticals, manufacturing, government, software, and financial services.
Keith’s career highlights include leading global initiatives to consolidate multiple data centers, unify disparate IT operations, and modernize mission-critical platforms for “three-letter” federal agencies. His ability to align complex technology solutions with business objectives has made him a sought-after advisor for organizations navigating digital transformation.
A recognized voice in the industry, Keith combines his deep infrastructure knowledge with AI expertise to help enterprises integrate machine learning and AI-driven solutions into their IT strategies. His leadership has extended to designing scalable architectures that support advanced analytics and automation, empowering businesses to unlock new efficiencies and capabilities.
Whether guiding data center modernization, deploying AI solutions, or advising on cloud strategies, Keith brings a unique blend of technical depth and strategic insight to every project.
