The Challenge of Creating a SuperCloud with Rob Hirschfeld

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

All right, joining us for another episode of the CTO Advisor is a multi-return guest. I've had Rob Hirschfeld, CEO of Rackin, on a few different times. And the theme of our conversations usually end up around platform teams. His product, Rackin, enables platform teams at the edge. But I think what is a common problem is what happens when we throw developer-centric problems over to, we can call it operations, we can call it platform teams, whatever you want to call it. When we throw it over the fence to those teams, what happens?

Rob, welcome back to the program. Keith, it's amazing to be back. I'm really looking forward to this conversation. It's something that's been top of mind in a lot of ways for years and years. But you're right. With platform team conversations finally starting up, it's definitely coming to the fore. Yeah, so let's give the audience some history around this. I like to give Rob free CTO advice. He's a CEO. I'm a CTO Advisor. So a CTO Advisor giving a CEO free advice, take it, you know, kind of with that warranty.

Like if he comes back and he says, Keith, this didn't work out, hey, you know what, you're a CEO, I'm a CTO Advisor, and your advice was free. So a point we want to make is that this is an output of a conversation we had way back at AWS reInvent a couple of months ago. I was Rob was showing me his product and talking about the transformation that developers have to use to make it. And I said, man, Rob, what would be really great is if you gave me a AWS CLI into the product.

That is the oversimplification of it so that developers, if they're AWS centric, they don't have to change their preferred programming interface. They can continue to use the programming interface they love. And as the application platform team or a support team, I can change the back end as much as I want. That's the theory. Rob, let's. That's the goal. Well, this is this is the challenge. Right. And, you know, I think from from my perspective, right, you get to whisper to my customers, CTOs.

Right. And tell them how to maximize the value of their investments, which is really what we're always talking about. And the challenge that we've had is for the last several years. And I don't see this stopping. We have talked to the application development teams or app dev teams. Right. So when we talk about developers, we're talking about the application development teams. And we've told them to be productive, be independent, go do the work you need to do. Right.

Operations get out of their way. You're just you're just slowing them down. Make it make it work. And in a lot of cases, what's that translated into is those teams using tools like Terraform to build infrastructure or AWS CLI to build infrastructure. And while that works really good for the app dev teams, when the CTO or really the CIO turns around and says, hey, wait a second, I need you. Operations are accountable for how all this infrastructure got set up.

Now that that team says, hold on a second, I don't even know what commands you're running to set up. Right. It's you're totally in the dark, let alone this idea that I'm trying to help 10 teams and they're all running it from their desktops or from their clients. Right. So so you don't you can't even tell what people are doing. So you might have somebody who uses Terraform to set up an infrastructure and then runs a script on their desktop to finish that job.

How do you know that? That's that. It's hard to put that toothpaste back in the tube. Yeah, we're we're going through these growing pains again when we're trying to perfect cloud based operations. We can parallel the platform team to the infrastructure team of 10 years ago. If storage costs got out of control, we could look towards our storage vendors to give us an innovation like deduplication. And instead of having each storage, each application team go back and rewrite code to optimize for the various storage realities, cost realities.

From an operations team, we could just turn on dedupe and magically we're using a tenth less storage than we were before because we've reduced the do the duplicate records on a storage array. Right. That is extremely hard to do in the cloud when one person uses Terraform to describe their infrastructure. No one uses cloud formations. Another one just does it by hand using the AWS CLI. And let's not get into multiple clouds. This is a single cloud problem and developers, different teams choosing different methodologies.

So I kind of came up with the brilliant original idea. I'm going to say no one ever thought about this before. Rob, why don't you just create a CLI proxy? Let's just for better lack of a better word, a proxy that would take those developers who want to describe their infrastructure or their infrastructure. They're rebuilding over and over again. They want to describe it in CLI. Why don't we just make the translation for them? This is a typical thing.

They'll get thrown over to the platform group. And the thing that you described, I want to explain what it what what we built, because I think that'll help people sort of figure out how all these pieces go. But the way I heard that use case is I want to take an existing API or AWS CLI call that I that I'm using in a code or in script or on my desktop. And without having to change that call, I want to be able to proxy it or control it through another platform or through for us through digital rebar and through the digital rebar platform.

And the key thing, one of the key takeaways here to me was that it needed to not change the API from it from a user perspective. So if I've written a batch script and I've embedded AWS CLI in that script on my machine or it's in a task in GitHub or a GitHub action or a GitLab task. Right. I want to be able to substitute that command without having to worry about the script failing. So it has to be able to have fidelity.

And this is this is a specific case. I think this is a universal example. And this is why we're having you on. This is why this is not a sponsored podcast, because this is a universal problem. I hear this over and over and over again. Right. And back then I didn't have the experience and language to explain why that's not the solution. It will help mitigate some of the problem. Yes, sometimes the answer is to buy more storage.

But other times the answer is to change the overall process. And the problem with changing the overall process is changing user habits, whether those users are developers or those users are in users, changing in user habits are extremely difficult. So when we can do it on the back end, it can be helpful. However, that was kind of our pie in the sky conversation back at reinvent. You went back and you built the thing. So this is and this is what's exciting.

So we built a script. It's a batch script that literally takes. H. Very clever naming. But it literally just passes whatever the parameters are into that script. S. H. discover instance easy to discover instances. And what it what it does is it send it creates a job in digital rebar, what we call a work order. That work order runs. We have a DRP CLI broker that just runs the DRP. Sorry, the AWS CLI. So it's an AWS CLI broker.

We pass that in as a task to run that command on the AWS CLI running in a digital rebar instance. And then from there, it runs the command using the credentials in digital rebar rather than the user's AWS credentials. So there's a proxy benefit here and takes those results, packages up as a return parameter and sends it back to the user. H. is calling the AWS CLI, but it's doing it inside of digital rebar. And one of the nice things is because it creates a work order for that, it creates a job for that.

It creates a task. It's audited. You could actually start adding different parameters and checks. You could say, you know what, you're not allowed to do anything but the EC2 CLI here, or you're not allowed to run these extra commands. Or if you run certain commands, I want to set a log or an alert or generate additional events. So all those things become possible around what would ultimately just be an exact syntax substitute for the AWS CLI. So people may be asking like, OK, I don't rock the value of this.

Let's let's let's let me kind of tell you why I asked for it. I'm not a user of digital rebar. I'm abstracting the way why I want this. Why I want this, because I think from a managed service provider lens, if I'm a managed service provider and I'm servicing multiple customers, AWS environments, and I get to the point where I have leverage, I have some skill and I'm getting some cost redundancies, I can now do one of two things. I can proxy all of those requests to my same AWS account and get the advantages of the reduced pricing without changing the interface for my individual customers.

Each one has their context within this platform, in this case, digital rebar. Each one has their context. And then I can continue that abstraction into my managed service provider offering or I can do something a little bit different. I can make an API compatible solution on premises. And now instead of servicing the request via AWS resources, I can if I have some Gen 7, Gen 8 hardware that's going to do the job and this is all, you know, basically free to me equipment.

Why not? And the from a developer experience, it's a it's a one to one experience. Is that thinking valid? You know, there's there's definitely the, you know, each cloud and every every system has unique syntaxes. So so it's very hard to make a universal AWS to something else translator. But but what what this lets you do is you can actually start looking at what the calls people are actually making are and then saying, oh, wait a second there. They need these actions.

Let me provide those actions. So there's an audit capability and there's also a there is an opportunity to rewrite the task and then redirect the task. But you have to have an API abstraction point to capture that. So what what you're describing to me is a very real use case of I'm running these AWS CLI and we're using this as a proxy. It's nice and concrete, but I'm running a action inside of a broader workflow. I want to be able to audit it, check it, redirect it if I need to conform its security.

One of the things that I get excited about is I could actually do a dev test prod type of work where the script we're using could actually be behind the scenes, redirected to dev instances or prod instances. Right. And you could actually have the same script now used in different cases, in multiple cases. And the infrastructure is code repeatability of your infrastructure starts going way, way up. All of these are important interaction points that are not easy to replicate from just the AWS CLI.

Right. Yeah. Or any lie. Right. And these are great insights. So let's, you know, extract away even from the digital rebar use case. I love this listening mode. You know, we look at the two hundred and thirty eight services that are in AWS. The reality is what the average organization might use 10, if not that. And of those 10, when you drill down and say, OK, I want to recreate some of these services or a subset of it.

Well, let's look at easy to, you know, of easy to am I using the different. I don't know. I don't know if it's a with the sizes, with a combination of size and type. If there's more than I think there's more than a hundred different size and type combination. I think I think they crossed over to a thousand just of instance types. So just flavors of instance types. Do I need to recreate every form of that instance type? Can I just proxy that these that basically we use 20 different easy to types and I can recreate those easy to types on.

Via my on prem hypervisor of choice, it doesn't really matter. I can look to see, OK, developers in their development environments almost always go to these three different types. I can do that on prem and not recreate a bill when they push it to production. They use these eight different types. We're going to continue to push those to production on AWS. But these other ones we're going to service on prem. That is kind of the goal. But as me and you discussed last week, there are unique problems because you get it to even in that limited set.

You get it to edge cases. You do. And one of the things that I think is interesting as a platform team or an operations team is coming in. You know where you can see where and how you can support people as they're coming into these systems is actually a really important component to consider. So there's edge cases in how things get mapped between different infrastructures. Right. You can't just take an AWS call and automatically map it into an on premises, call a bare metal machine or a VMware machine.

Those those mappings do require a certain amount of understanding of how they work. But one of the things that I find is just how do you answer the phone when somebody says they have a problem and can they replicate that? Can they replicate the challenge? Right. One of the things that's interesting about edge cases is that a lot of times an edge case ends up being something where you need real expertise or support to help resolve what that problem is. And so part of the value in this is you actually want to be able to get your experts on the line, have them duplicate exactly what happens.

See the logging, figure out what what went into that circumstance and then ideally replicate it. And even more ideally provide guardrails like you've been describing for the next pass on this. It'd be really easy to write a series of commands that can't execute because somebody doesn't have the right permissions. They're not in the right VPC. They don't have the AMIs that they're using for the machines don't exist in the regions that they're trying to execute them on. Right. There's huge numbers of failure possibilities and how all this stuff gets gets fit together.

And I think one of if you're listening to this and you're thinking through, wow, I could proxy between cloud providers. One of the terms we use in our conversations is super cloud. This is the beginnings of creating a super cloud. Developers all develop in AWS, CLI or Terraform. Pick your preferred language to describe your infrastructure's code environment. You settle on one form and then that form can be reapplied across multiple cloud providers on the platform group site. You can now proxy and say, oh, I can intercept a request that was going for AWS and now I can send that request to Azure.

That was how I was thinking from a market texture and drawing this out perspective. But we got into some of the details about the talk to me. What happens when I request the AWS machine from Azure and we proxy that request? You know, it's interesting because at some level machines are machines are machines, which is which is, you know, thankful. I'm thankful that that when you're running a Linux machine, you know, of a certain flavor and OS distribution, that OS distribution usually translates pretty well cloud to cloud on premises, off premises, bare metal edge, wherever you are.

There's certain certain components that you can take for granted once you get into a certain OS. Unfortunately, we're still in a very heterogeneous world where if you're using Amazon's Linux, it's different than Oracle Linux and CentOS Linux than Ubuntu Linux. Right. You know, you can't get install can't even get installed across all of the Debian distros anymore. So we do have some real we do have some real differences. You know, and this is this is the challenge. I actually like to describe this as a fruitcake instead of a layer cake.

People want people want nice, clean layers like, oh, the storage layer is different. It's the same in the OS layer is the same and all that. But it doesn't actually work like that, like how a disk gets set up or network gets set up actually has to be accounted for throughout the layers above that. You actually have to account for differences between the systems. But but to your point, if I can say if somebody says, give me a Alma Linux machine of a certain generation, then I can get that from pretty much anywhere.

Put an SSH key on it and give it back to you with. And you can say, I want this much RAM and this much storage. And you can abstract that successfully across pretty much every platform that you that that we encounter, which is every every environment. The challenge becomes now we're straight back to our proxy case. The way you ask for that and the syntax you use to build that generic unit is highly variable across the across the clouds. Even something as simple as machine ID.

If you inventory off a machine ID, if you create some type of sock pipeline based off of the machine IDs, you've you've kind of figured out the identifiers within the AWS machine IDs and you build some process on it. Well, an Azure machine ID is going to look very different than a AWS machine ID, which looks different than a G and then the GCP machine ID. The even how you figure even how you figure out which which machine ID you have is variant across.

I was just about to say, because the machine ID field may not even be the same field across cloud providers. So that metadata problem becomes a operational issue because these unique identifiers just operationally, we guys, we optimize based on the on the things we can optimize on. And machine ID ironically becomes one of those things that become critical. Unintendedly. Yes. So, Rob, we talk about this stuff all the time. I was just talking to a young CTO and he was saying that he just doesn't have relationships.

And I'm thinking, man, Rob used to run this this weekly thing called Cloud 2030. We're surprisingly getting a little close to that. Might need to push out to if people want to engage with, they want to like bounce ideas off of you. How do they find you? How do they engage with you? We are still having a weekly DevOps lunch and learn roundtable. And it really is a roundtable. People bring topics and we discuss it. And I love the format because we spend 15 minutes completely off agenda.

And then usually we'll try and jump back to some some planned topic of the day, which is some Kubernetes or platform engineering or API design, classic DevOps stuff. People can find out more about it at at dot the 2030 dot cloud. So it's I couldn't get the cloud 2030, but the 2030 dot cloud. We post the agenda for that. We actually also do a broader strategy breakfast where we talk about much more esoteric topics like metadata context for artificial intelligence, data gathering and the future of Internet regulations.

Both are important, but they're great places to engage and ask questions like this. Very pragmatic. And generally what I describe is two or one level type stuff. So not, you know, how do I use Terraform? But, oh, my God, I'm using Terraform and I've gotten to a point where it's really hard and difficult to scale because, you know, I'm storing the files in different places or I've got 100 copies of the same plan and I don't know what to do. That those are which are very real, right?

Everything we're talking about is an acknowledgment that cloud complexity is real. And, you know, you have to be deliberate in how you scale it. But that's the cloud 2030 conversations. Go go go to that. A lot of fun. Very casual. Yeah, and I don't attend as many as I want. But every time I attend one, I learn something new. It's a eclectic folk from the super technical like Rob to the less technical like me. And it's a really, really great conversation.

If you want to learn more about the CTO advisor, you can follow us on the web. The CTO advisor dot com. You can at me on Twitter at CTO advisor. I'm just looking at Rob's feed out his window and it looks like he's getting a package. So Rob, we'll let you go. Talk to you next episode of the CTO advisor. Thank you. It's been a pleasure.