Accelerating ARM and x86 CI with Actuated

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

All right, you're listening to another episode of the CTO Advisor podcast. We have a return guest, my good friend, Alex Ellis, who took time out of his busy day of building crafts and watching babies to join us to talk about CI on ARM. Alex is the founder of OpenFAST LTD. S. because otherwise it would be the LLC. Alex, welcome back to the show. Hey, Keith. Yeah, OpenFAST Ltd. And it's as a as a company mainly represents OpenFAST.

But now we've got Actuated as well. We're going to be talking about today. That's a new thing since we last talked, what is Actuated? Yeah, so we had customers on it for about 12 months now. And then going back probably six months before then was a lot of R&D I was doing with Firecracker, basically seeing what it could be used for. They've always liked containers and virtualization and being able to spin up servers and do stuff with them, tear them down again.

And Firecracker is a tool from AWS that they use for running Lambda functions and for Fargate containers. And it's designed for short-lived ephemeral workloads, which sounds a bit like serverless. Right. And so the reason that I was looking at that is I was maintaining extra servers, bare metal servers for all of our CI needs for OpenFAST, for the software and other team developers, etc. ARM machines, x86. And it took me a long time to set them up. The packages would often conflict.

So if one build needed one version of a package and another build needed a different one, they couldn't run at the same time. If they ran after each other, we had to clean it up. And there was none of that with a hosted runner, because on GitHub's hosted runners, you get a clean Azure VM every time you do a build. There's no side effects, no software to install. And whatever you do in there, it's going to be gone by the next time you get in again.

And that's the experience we wanted to recreate. How does the CI process get impacted if you're using, if you're developing for or on ARM? Yes. So the product that we've built is for both architectures. And there's a third one that hasn't really hit mainstream yet, RISC-V. That's still super early. But if you look at what's happening with RISC-V, it's very similar to what was happening with ARM, perhaps as early back as 2015 or maybe even earlier, which was the you would want to perhaps run on an ARM device and there'd be packages missing, upstream projects without support, even operating systems and basic things like Docker that weren't available, that you'd have to build the world basically to do this sort of stuff.

Today, with the popularity of Apple Silicon in particular, and all the work companies like Ampere and ARM themselves have done around enablement, you can take some software that would normally run on an x86 server and run it almost exactly the same on an ARM server whilst getting benefits of better power efficiency, much higher density of cores and numerous other benefits. And so when it comes to CI, it's very, very similar to a normal x86 workloads for actuated customers. But for GitHub today, you can't build ARM on their platform because they don't have the VMs, they haven't integrated into their system.

And so you end up having to use an emulator called QEMU. And I don't know if you've ever been here, you only tend to use QEMU the first time you've got to build for ARM, call these maybe multi-arch Docker images, for instance, where OpenFAS will run on a Graviton or an Ampere Ultra and also on a regular EC2 instance. So we build for both architectures, we publish one image and then when that's running Kubernetes or run on a just on a regular Docker machine, the computer pulls the right version of the software down and runs that instead.

So that QEMU thing is just ridiculously slow, that's the kicker with it. So it sounds like actuated, if you're looking at it from like the developer operator perspective, this is an all-in-one workflow. So if I'm a pure operations group and I've, you know, I've kind of built my own alternative or workflow to not necessarily create packages specifically for distribution. But if I'm, you know, doing my own CI process and I want to dynamically select where I work around the workload that I need something like actuated or a process like actuated.

Yeah, so what we're providing with actuated is ephemeral, immutable VMs for CI. And then what you do within that is entirely up to you. GitHub Actions really is a CI platform, but you can run other stuff in there if you want to. And so some of the customers, yeah, some of the customers that we've adopted like Toolpath, for instance, they build code with Julia and they were finding that their build process took six hours of compute time every PR and we basically knock that down to one hour.

Because you're building on essentially, I won't say bare metal, but native VMs that run on ARM or bare metal ARM itself. Yeah, so in this example of Julia, I believe that was all x86. But the speed increase we got there was by moving to the best in class, fastest processor with DDR5 RAM, NVMe locally. And then these micro VMs are tiny, like they don't have PCI devices, that's not even built into the kernel, it boots directly into the init. And in less than one second, it's even started Docker.

And so whatever you want to do in there, it's like a hermetic environment, do what you need to do, get your artifacts out, ship them to the customer, ship them to production. But where we see the most dramatic speed ups are with ARM. And it can be with an open source project, it could be a commercial one. One of the commercial ones I can think of would be Waffle Labs. And they had this situation where the guy who was the operator had set up a self hosted runner on a Graviton instance, he had to babysit it, he could only run one job on there at a time.

And so the team was backing up and was really frustrated with it. It was very fragile. We moved them over to Actuated and a server from Equinix. And they've been super happy ever since, nothing but good things to say, and their build is a lot quicker. One other example just quickly of an open source project, because you have to say not to run the self hosted runner for open source projects, because it's not secure. You can leave side effects, people can potentially take over the account, install malware on the host, etc.

So Ellie from Atuin has this CLI tool that syncs all your bash history encrypted in the cloud. So if you talk about going from your, I don't know, let's say like your desktop over to your Mac, maybe use the Mac for traveling, all of your bash history would follow with you. Or if you went to a different like a Linux device somewhere else in the cloud, and you had Atuin installed there, it would sync it all. Now she had a Rust binary that they were building with QEMU that I mentioned earlier, this emulation that took 90 minutes.

She tweeted that it went down to three minutes by adopting Actuated. So talk to me how Actuated is deployed or delivered. Because in my mind, if I'm a platform group, I want to run this on-premises, I have my own bare metal, I have my own VMs, I have my own AWS account, and I can just deliver it. On the opposite side, if I'm used to using something like GitHub Astions, this is a lift while I benefit from the speed reduction, while I have to lift to actually administer the platform.

How is it delivered? Do I have both options? Well, you talk about this lift. Well, if you were to just try and use a self-hosted runner, that's where you get this lift, this friction. And if you try to use one of the open source controllers that runs on Kubernetes, there's numerous issues with that. There's a huge amount of administration. To run it securely, you need to set up and look after a completely separate Kubernetes cluster. And if you use Docker at all, it's going to use a privileged container, or it's going to mount the socket from the host, or do any of these kinds of things.

They are huge security malpractices. And really, a lot of people do them because they just don't know any better, or they think it's the only option. And what you end up with is potentially a completely compromised system. With Actuated, what we've tried to do is match the experience of hosted runners as close as possible. So, we do all of the integration with GitHub. We provide a dashboard, reporting metrics, some extra features like SSH debug for when you just can't get your build to work.

And then all you do is bring a computer along that has KVM enabled, install an agent, and that's it. Once the agent's installed, we remotely monitor it. We look at the load. We look if it's up or down. We look if it's over-provisioned, under-provisioned, and we can tell you how things are going and have a full level of support with it. So, basically, you're not providing the compute itself. You're providing the control plane for Actuated and CI in general.

Similar to something like a Platform 9 or one of these other systems where the Kubernetes or control plane resides in the public cloud. And I push down an agent to whether it's my bare metal x86, bare metal ARM, VM, EC2 instance. It really doesn't matter just as long as it's a Linux host. I put the agent on, send my package up. The Actuated orchestrates the build of the package. And then if there's an error or whatever, it throws back to the dashboard.

But basically, my compute, whether that's local or in the cloud, is just dumb compute for doing the actual build. I mean, that's what makes it so simple. And Actuated isn't even as complex as you described there. It basically has one job on your compute, which is find the best candidate for scheduling the VM, start it with a secret token, and then the VM itself will just start the same runner that is unsafe to use otherwise in this hermetic environment that's immutable. And Actuated doesn't get involved with the code.

It's not got access to the code. It's not running on our server. It's completely within your hardware. And at the end of the build, it reboots the micro VM, which is the same as stopping it. And so that's effectively why it is so simple and why we've been able to roll it out to do over 140,000 builds now with largely no issues. So from a communications perspective, let's talk about kind of that security. When I'm pushing the code, where's the code?

Where's that communication channel? Is it from my desktop client where I'm developing the code, my code repository straight to the KVM instance that I'm dictating, or does it flow up to the cloud and down? So the first thing that we should say is Actuated isn't building your code. There's no awareness of what is in your repositories or even necessarily what is going on inside that VM. Its only job is to receive webhooks from GitHub that say that a job is needed.

It looks at the labels to see if the labels are one it looks after or not. If it is, it would then use a scheduler to inspect your fleet, find out the best place to put the micro VM, obtain a secret for you that can only be used once, that the runner will use to register itself, and that's it. Like it doesn't know anything else beyond that. And then if the build fails, well that just goes back to GitHub in the GitHub UI, and with our dashboard we just provide some visibility of how many VMs are running, how many servers you've got, the capacity, and then something that GitHub doesn't do is organization level metrics.

So we can show you who's the heavy hitters on PRs, who's got a lot of PRs that are failing, which repos are getting the most action, the longest build, average build time, etc. And we can do that across the whole month and contrast it to the previous one. So you can look maybe as we come into Thanksgiving and the festivities, you'll probably see your month by month comparison on our dashboard take a real nose dive. So from a capacity and performance management perspective, as I'm thinking through whether or not I either move resources from a different pool of compute to my CI capability so I can gain, you know, additional agility, how do I get insight from the platform to know, hey, you know what, if I either went with a bigger EC2 instance or added more RAM to my ARM instances, I will get better performance, I've gotten better performance.

So ultimately, the main thing that happens when people adopt a platform is they'll see at least a three times improvement and they're just so happy with that, they just move on and get on with their work. What we can do is we've got the load averages of every server in the fleet. We can let you know, for instance, if your server is dramatically underutilized, you might be able to get a smaller one, or if it's really being pushed. So we look at the load average and the load average often correlates quite nicely with the amount of threads in the machine.

So let's say you've got the equivalent of 32 threads, like a modern Ryzen 16 cores. If your load average is going over that, chances are you could benefit from a second server, so you can spread your jobs out more, or perhaps, you know, you're running too many VMs on there, you need to reduce the core count, and there's various other things that can be done. But really, this, for most people, is a plug-and-play experience. You come on, join us Slack, we'll often ask you what your previous build times were because we want to see what the improvement is.

We'll recommend a server there and then, or you can look at our docs to find one that fits your needs. And then, if people don't see an improvement, they obviously won't keep paying us, right, because this is a product, a high-value product, a very technical product to deliver. And so we try and make it as simple as possible, support people as much as they need, mainly around their onboarding experience. And then, for the most part, we don't hear a lot from people other than, you know, like good things.

There is a mixture. So one of our customers buys refurbished servers, puts them in their own data center, and they run all of their builds over three large bare metal hosts. We have others where we'll refer them to places like Hetzner, and they'll get a super fast Ryzen 9, and they'll see even something like a five-minute build, which you might think, oh, that's great, come down to one minute. And when you're pushing PR after PR, or you've got, you know, five, six people in the team pushing code, going from five minutes to one or 15 to three is quite a big difference in that experience.

So have you gotten, you're a fairly new project, so two questions. One is on compliance. Have you gotten any customers who've been audited since deploying the solution, and have auditors paid any attention, or they're just not that far deep in the weeds? So we have had one company ask whether we're SOC 2 compliant. And the thing is, you need to get to a certain size, or you need to perhaps be a VC-funded company to go and do SOC compliance before you have product market fit, and before you have your first, I don't know, 500 customers.

So it's generally not something that is that important for the current suite of customers we've got, but I can certainly see it becoming more important as we get more and more people on. I think one of the things that helps mitigate concerns is that you can run the runners on your own machine. Our application has no access to your code whatsoever. It doesn't have the API permission to clone your code or look at it. And so really, that compliance then falls down to the GitHub runner, and GitHub is SOC 2 compliant.

This would be actually, it would be extremely interesting to my audience, this ability to say, hey, we're just going to take the code from our developers using our selected automation tool, and our selected deployment tool, run it through, use Actuated to compile the code for the platform that we desire, because we're switching from, there's the desire, I don't know if there's actually customers trying to do this, but there's a desire to do this. Switching from x86 to ARM to chase the best performance. One of the pieces of research we're looking into is taking a deep dive at looking at Graviton versus x86 in general, and the lift associated with moving code from one platform to another.

Yeah. This seems to be a potential solution to that problem. Have you gotten customers that want to do portability between platforms? Well, that's definitely a concern for Das Swiss Data Center for Humanities in Switzerland. They build code for x86 and for ARM because they don't know where, what they distribute is going to be used, and they do have customers that deploy to Graviton, especially v3, which is the latest generation. If you're doing any benchmarking, we benchmarked Graviton 1, and it was incredibly slow.

It's one of the cheapest ways to get bare metal on AWS, is the A1 metal, the original instance. But if you compare that to their latest generation, the v3 just wipes the floor with it. And so you've really got to look at what is the clock speed of the processor? Is a single core, multi-core? What is the RAM speed? Have you got NVMe locally installed? Is this a normal SSD? Is it spinning rust? There's a few factors that come into it, and it can even come down to network speed as well.

If something is maxing out the system, if you've got a one gigabit connection, or you've got maybe half a gigabit, you're potentially going to be twice as fast. One of the things that I mentioned earlier on is the security aspect to this, and how for certain of our customers, having open source repos and knowing they're being built in ephemeral environments for ARM is really important for them. For the speed increase, for accessibility, it's much nicer to contribute to a project where it doesn't take one and a half hours to get feedback, but three minutes.

And so Ampere, who make the Ampere Ultra server, this is available on Azure, Google Cloud, Oracle Cloud, Alibaba Cloud, and potentially a few others in a VM format, on Equinix Metal as bare metal, and Hetzner as bare metal. These servers are just becoming the place and way to get ARM today. So they approached me because we'd been talking to the CNCF about providing Actuated for a number of projects that are having these same problems, slow builds, worrying about security, or having teams maintaining servers.

And so we've signed a pilot deal for the first year, and we're going to get a number of projects onto that. The first one was Fluent Bit. They provide a log aggregator that some of your customers and your audience might use to basically collect logs, traces, metrics, system information back to centralized point to manage all their servers and infrastructure. So that project team, they were running to six hours because they were using QEMU to try and emulate ARM instructions. And six hours is where GitHub Actions actually cuts you off.

So they just couldn't get that build done at all. Now Patrick Stevens has been really complimentary about the product. He's gone spoken about it. We basically got it down to five minutes. So from them not knowing if it would pass in six hours, it finishes in five minutes. That's now under the CNCF deal with Ampere. And the second project we got across was at SETI-D, which is the key value store inside Kubernetes that has all of the state so that all the other components can effectively be stateless, restart, scale up, scale down.

It's almost like a database, if you like. That is now building on Actuated as well. And we're currently talking to Cillium to get their projects brought across. So the CNCF and Ampere definitely see this as a problem that needs solving. They see value in that ARM platform being available for people that either want to run an IoT device, maybe at the edge, or maybe they want to run on an Ampere server, or maybe they want to run on a Graviton if they're AWS customers.

That's really interesting. And I really love the insight from Graviton1 to Graviton3. Some of the research that we're doing is Graviton1 to Graviton2, and then 2 to 3, that migration process. Optimization of code is another topic that, man, you can discuss one day. But it's not as straightforward as a process as you would think. So you folks are doing a lot of the heavy lifting, a lot of the behind-the-scenes stuff that we really don't think about. It's new to me that people are actually using QM in any kind of effective way because it's not a platform that I've played with for probably 12 years because it just wasn't, one, it's not a great virtualization platform in general.

And then two, emulating, doing any level of emulation is not typically production stuff. We would use it to do risk, previous versions of risk to x86, but I haven't ran it to too many people in my life that do it for ARM emulation. So it was interesting to see that the project is still kicking strong but not probably fit for this case. I think QMU, some of the time, is okay. I have a few open-source projects I maintain where we use it because it's free, it's easy, it's available.

That's why most people start with it. But as soon as you bring in a dependency or two or try to run some end-to-end tests, you go from 3 minutes to 30 minutes to 3 hours. It's like an exponential gain in time, and it might just be, oh, I just want to run some unit tests in Go. And now you have this huge issue. So we get around that by using the native platform. And then we get around the security and immutability problem by using an immutable VM disk with Firecracker.

Firecracker doesn't even have a PCI driver built into it. It literally has storage, one key keyboard. It doesn't have any GPU support. So it's very, very lightweight, comes up really quickly, a great API. And some of the stuff I've been doing over the last 18 months is just telling people how to use it, talking about different use cases. And we actually had a separate scenario come up recently with OpenFaaS and a customer that was getting to 3,500 functions, and then they saw a 10-minute delay before the next one would come into the system.

And I figured that I would need 30 VMs on AWS or Azure to reproduce a problem, which would cost me $1,500 for the duration of the test. And so I was actually able to use Firecracker to slice up my own bare metal machine, my workstation, and the Ampere machine I have behind me into 60 VMs. So double the amount, found the problem, fixed it, and then was able to test up to 15,000 with bringing a third machine on and splitting that up and having, I think, we had like 120-node cluster all in VMs.

And Actuated is a cool product. It's really helping people. People are paying for it. That's the business model. You pay if you think it's valuable to you. But Firecracker's technology has so many different use cases. Yeah, we've been a fan of Firecracker for quite some time here. Micro VMs in general, the ability to run basically stripped-down VMs with the minimum kernel, speeds that not quite as fast as containers for spinning up, but close enough for most people. And you get the full security of isolated kernel.

There's a lot to like. Alex, I'm going to leave you with one bonus question that has nothing to do about what we talked about because I just haven't gotten the chance to talk to you about. And this is a related topic to the whole Tofu versus Terraform stuff that's going on in the open-source community. I'm coming out of HashiConf. It's probably been about three weeks now, two or three weeks since I went to the show. Not a lot of talk, surprisingly, at an enterprise IT conference about open-source projects that are emulating what the proprietary software vendor is doing.

Surprise, surprise, it's an enterprise IT show. And I think people forget that about the folks at HashiCorp, their enterprise IT shop. What has been, amongst your peers, you run an open-source project, a pretty popular one. What have been the feedback from your peers about the friction happening in the open-source community? So, I mean, when you talk about my peers, I'm a business owner, an employer. I'm trying to solve problems and make money at the same time to pay payroll. Now, not just that, but I want to grow the company, add more people, have the revenue to back it up.

So, we've actuated. Nothing about it is open-source. It has an open-source CLI, the docs, website. It's a hard lesson to learn that people don't pay for what they can get for free, especially in the context of open-source software. And nowhere have I sort of learned that more than with OpenFaaS and the last six, seven years of building that. So, I think there's a lot of people that are ready to share opinions and come in, you know, in their armchair position and criticize unless you have run an open-source project and tried to make payroll at the same time.

I don't think any of us are really entitled to have an opinion here until you have done. And then I think you might find it's more nuanced than people let on. Well, I really love that answer. Very diplomatic and I think very accurate. I've watched your journey over the past few years. I'm really happy for the success of Actuated. If people want to learn more about Actuated, how can they find you? Yeah. So, if you want to look at the CNCF announcement or the docs or like you said, how does security work?

dev or on Twitter, we have the account at SelfActuated. And very easy to find on the internet. Go have a look, poke about, ask us questions, look at a case study. I'm very happy to talk to people who might be interested. I'm glad you found the model that works. com. We're now a Futurum Group company. We'll continue to do the integration on the backend. com. Talk to you next, CTO Advisor Podcast.