Twitter Scale with Bobby Dorlus

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

All right, you're listening to another episode of the CTO advisor podcast. We've been trying to get Bobby doorless on for about a year now. Schedules have not, uh, lined up, but fortunately he just up and quit a great job at Twitter and now he has all kinds of time, Bobby, welcome to the show. Thanks. Thanks, Keith. I really appreciate you inviting me. And like I said earlier, I, I have a lot of time. That means my mental capacity is wide open.

Definitely. So having a engineer of your quality and, uh, kind of resume is dangerous because I have a lot of questions and we typically only have about 20 minutes on this podcast. So I promise podcast listeners, we will not go over 20 minutes. If we do, you'll enjoy it, but we are, we will try not to, uh, go there. So Bobby, we have a pretty good shared experience. Both of us are people of color, specifically black men in, uh, enterprise tech or high tech.

I was just at KubeCon as this is being recorded. I was at KubeCon EU last week. No surprise. I only saw maybe four, maybe five other black guys at the whole show. Tell us about your journey. How did you end up here? Roger that. And, and Keith, thank you for much, very much for calling that out. Um, that's something that I noticed as also when I'm going to conferences, um, is, uh, the, the lack of, uh, diversity in terms of people that look like us, um, and it's an area that I focus on and definitely, um, we'll have some more chances talking about it.

Uh, but for me, uh, it's, it's been a struggle, Keith, uh, because, um, I come with 20 years of experience, but when people beat me, they still think that I just got into the game where they haven't seen anybody like me before. Um, and it was a struggle at the beginning and it's still a struggle. I think our industry is working towards a more diverse culture, but there's definitely room for us to grow. Um, so I'm originally from South Florida. Um, I'm a child of an immigrant.

My parents are from Haiti. I'm a first generation Haitian American. Um, but with that said, I still am proud, proud, proud, uh, to be somebody that has had the opportunity to, um, not go to college, I don't have a four-year degree, um, and really take certifications to the next level and really take my skills in an environment where you can be self-taught, um, with technology and engineering and level it up to a point where I work at Twitter, uh, which is my last gig, uh, for nine years as a staff engineer.

So, uh, it's something also in terms of just our environment, not to see as often, which are staff engineers in the Fang or, um, Silicon Valley space. Uh, but definitely if I can do it, you can definitely do it. And that's more or less what I want to make sure I ensure to any listeners that are on the line now is, uh, your background can be extremely diverse and you can still achieve great things in the long run. So, uh, that's how I got here.

All right. So I love the, um, uh, observation, uh, that you spent nine years at Twitter, basically self-taught. I was self-taught as well, starting out in my career. And then I went back and got a couple of degrees. Uh, don't regret it at all. Uh, I think I'm better at what I do today because of it versus being my potential to be a engineer. And it's pretty interesting that you're at a point that you have 20 years of experience and you're a engineer, you're specifically a staff engineer.

And for folks listening to this podcast, we get a point of reference. We understand, you know, the difference between, you know, like an associate engineer or engineer and the staff engineers, quite a big deal deal. What has kept you passionate enough to reach the level of engineering or, um, uh, uh, uh, professional level engineering that you're at today? Yeah. I mean, for me, I'm, I'm somebody that from the beginning always enjoy solving problems. I'm a problem solver at heart. When I was a kid, I used to take my toys apart and my parents didn't really understand what I was doing, but I just wanted to know how the thing worked, right?

I wanted to know why, when I press this button, the light turns on and the car moves. And that basically forced me to have a very, you know, obsessive passion to learn. Um, learning for me has been the foundation for how I've been able to level up. And one thing that I've also learned about learning is that, uh, this is a nugget Keith, is that our knowledge that we gain at every place that we've learned is the real asset itself. Um, and because of that, I've been able to take assets from each place that I've learned and taken them and applying them into the new job or new opportunities.

So working at Twitter, getting to a chance to work at Twitter, I brought and came to Twitter with at least 10 years of knowledge, so 10 years of assets. And I really found out that a lot of the things that I've done in the past were re-implemented, being re-implemented at Twitter and, or I was the first one to explain how certain things work. Um, these knowledge that we gain and the asset that we build over the years, working at different places, um, really allowed me to look past just having a college degree or, um, having challenges of being, you know, a diverse person in this industry and really put on the table, the knowledge and, um, that has really allowed me to ascend to, um, become a staff engineer and taking this to the next level.

So learning to learn, always learning, um, and just continue to be passionate about, um, how far I can go and what, what I can be exposed to in terms of technology, because Twitter scale wise, which I'll hopefully get into is it's extremely massive, bigger than I even thought. Um, and probably, you know, um, larger than most people have expected and also the actual efforts of engineering at Twitter. So achieving that title staff, I tell you, it was a big celebration because it was only one of a few of us that got it, especially if you look like me.

Um, and I was very much honored to, to, um, um, continue to, you know, contribute at that level when I eventually was promoted to that position. So let's talk about Twitter scale. What, what does, what does that mean? I mean, I've, I've heard, I've seen the meme like Twitter scale. Okay. Give, give us a sense of the environment. Definitely. Definitely. So, uh, one thing that, uh, I've observed through this last month, especially with, uh, some of the acquisition conversations that we've had going on, um, is that some people think, you know, Twitter is just an Excel spreadsheet, uh, or maybe a, a, my SQL database with a few, you know, uh, columns and rows in it.

But Twitter itself is a massive infrastructure. Um, it compares to our peers in terms of, um, when we purchase hardware and we're designing system, our peers are Google, Facebook, Amazon. Um, and really those hyper scale infrastructure because Twitter scale is on the back of hardware and infrastructure that Twitter builds. So we don't run much in the cloud. We build our own data centers from the scratch. Um, and we build hardware from the scratch. Um, and that right there just gives you an idea of why we would do that is because it just costs too much for us to go and buy them at HP Dell or any of the OEM providers out there and that reducing costs, which is building ourselves, emphasizes how large our scale is now, um, just that gives some numbers and association with it.

Um, so the team that I worked on prior to departing Twitter was on the compute platform team. This is a team that basically manages Twitter's cloud. So it's like the team that manages GCP. We're basically that team in-house and that team itself, um, of about five to 10 engineers was managing a scale of millions of cores, millions of CPU cores, um, and one of our clusters, we ran over 500,000 containers. Um, and we have more than one data center. That's at that specific scale.

Um, no counts range in the a hundred thousand range, um, and all in general, um, just think of between a hundred to 80,000 is a data center footprint. Um, and we basically have more than that in terms of just scale wise. Uh, we, we have this way of saying that we have N amount of data centers. So basically take that a hundred thousand in it basically, uh, for redundancy multiples of two threes. I can't give exact numbers, but that's kind of like, uh, the idea of our footprint.

So it's in the, definitely in a hundred thousand range of node count, millions of cores and running hundreds of thousands of containers around the world. So I love the breakdown. I've read a few publicly traded or public blog posts, not publicly traded. I'm having in my mind, all of this acquisition rumors at the, you know, the, the, as of this recording last night, there's a rumor that Broadcom wants to buy VMware, so my mind is a little scattered, but, uh, I've read some posts from Twitter engineers, uh, about, uh, the complexities of managing things at scale, uh, and the advantage of using, um, things like DPUs or smart NICs to, uh, reduce trail latency, tail latency, et cetera, et cetera, these are really tough problems.

Can you give us a, an example of a problem that Twitter would have to consider that maybe a typical data center enterprise would not even have to worry about? Yeah. That's, that's actually a really cool, um, question to dive into, uh, because I'm, I'm going to actually talk to you about a project that I worked on right before I left, um, and actually spoken about this on stage a few months ago. Um, it's the idea of running co-located workloads on the same bare metal machine.

So one of the challenges you have when you run at scale is number one, differences are anti-patterns. So when you have different hardware, um, imagine you have hundreds of different types of hardware. That means you have hundreds of different ways that that hardware can break, right? Or thousands. But when you have very light hardware, uh, you do run in a case where you can reduce operational burden because the hardware is all alike, meaning the operation team supporting it, the software managing it, so on and so forth.

But what happens when you have all of your services running on all the same hardware style and platform, but the service can't take full advantage of that hardware. And the example I'm going to mention to you is we had maybe half of the data center footprint running storage platforms, storage services, storage services, unfortunately could not take full advantage of all the computing resources inside of one computer that we allocated, which is the default computer. I'll give you an example. It's a Intel processor, dual socket, and it had about 80 cores.

The storage platform can only take advantage of maybe five to 10 of those 80 cores. That means we have 70 cores inside of data center, not using and being used for anything. And let's take that to the next level where it's not just one computer that looks like that. It's tens of thousands of computers that have that same challenge. With that in mind, designing systems at scale requires you to also unfortunately have some setbacks and inefficiencies. And the advantage of this also is that, hey, engineers like myself can go find ways to take advantage of those inefficiencies by designing systems to run co-located on the same bare metal machine.

So in normal environments, you may not have a challenge with one machine, you know, only using 10% of its computing resources, but when you have 10,000 machines only using 10% of the computing resources, just think about the amount of cooling and power that is being wasted. Those are the type of large scale, hyperscale infrastructure challenges that we've, you know, dive into to solve those challenges and gain those inefficiencies in some form or fashion. And it was wrapped up in that co-location project that I mentioned at the beginning.

So when you think about scale, it's one little thing could be multiplied by 10,000 times, and then that one little thing ultimately could be inefficiencies in power costs that you're saving the company by solving those types of problems. Yeah. I love that example because it resonates even in small data centers. The scale of the problem isn't as bad. You know, when we introduced a hybrid converge to the data center landscape, we introduced a similar set of challenges, but just not at scale, but we can relate.

IE, if you're using a, what Wikibon was calling server sand back in the day, where you're doing exactly that, their storage nodes, one of the problems is what happens when you need to expand your storage capacity, as you expand your storage capacity, that inefficiency grows, correct? So now when you're getting to hundreds, thousands, or tens of thousands of storage nodes, that inefficiency is kind of blatant. So I want to dive into kind of, uh, not the solution itself, but the solution the mindset that Twitter and you look at, as you look at a problem like that, like I'm really interested in the details of how, you know, how you solve the problem, but we'll, we'll stay high level on how you look at solving the problem versus how you solve the problem.

So obvious challenge is that in order to co-locate applications, you have this contention that there needs to be data to process if the nodes are storage nodes and their primary purpose is to serve up workloads, I think there is a resource contention problem of how do I optimize for CPU use a system that is primarily primary, most valuable asset, the storage IO, I might have to borrow some of that. So how did you look at that problem and go about approaching solving it? Yeah, definitely.

Um, so one space that actually came available to us is in our process of designing new hardware throughout the years, our hardware engineering team really worked closely with Intel to get us to a CPU set that has the capability of solving the exact problem you're talking about. Now, when we're designing these systems, I'm talking about on the motherboard PCI, where is that connected to? How does this drive connects in terms of up to the CPU? And what's that path? The design strategy that hardware engineer came up with is that one socket would have that PCI path going directly to, um, the hard drives and then the other socket would have a different PCI path to other devices that may need to be used inside of that specific computer.

Now, what happened is, is that what we did is we said storage services. When you run, I don't want you to go onto the other CPU because your path will introduce latency for you to get access to those disks. We want to make sure that when you run your pin to core number zero, because core number zeros pass with the PCI all the way down is connected directly to your NVMe drives. That right there was the key for us to introduce isolation because these nodes themselves, like you said, are dedicated for storage workloads.

As long as we can get the storage nodes to access the CPU, no introducing into any latency, and we can isolate those workloads to a certain amount of cap, then they, them themselves are still serving their customers the same way that they were servicing them before. Now, once we introduced that cap, that means that we had one whole socket in terms of a actual hard core socket with its threads and cores that was sitting there, not doing anything. Mesos and Aurora's main computing resource that it's looking forward to is CPU.

We don't store any type of persistent data, CPU, memory, and disk. Soon as we say pin yourself for storage units to that socket, we said any workload that's Mesos and Aurora or a compute workload, pin yourself to the second socket. When you send your traffic, go through that PCI slot or that PCI path to that network interface, and that's a separate isolated component compared to the one that's servicing storage. That right there created our separation so that tail latency on the storage side actually got better because they didn't have to traverse the interconnect between the CPUs to access disk, and they were just pinned specifically to the socket that had access to those PCI devices.

And ultimately performance actually increased with this implementation, and then we were able to take advantage of those resources that were unavailable. Wow. So let's end the conversation talking about that whole Mesos approach versus Kubernetes. Like if I'm, if I'm a data center architect, if I'm a cloud architect and I was told to build a cloud platform, you know, right now the de facto option is Kubernetes, like that's just, you know, why wouldn't I do Kubernetes? And the question that I want to ask is more around how should people look at that journey versus just saying, oh, the de facto standard is Kubernetes, so I should start there.

Is that where people should start or should they start somewhere else in the analysis and selection? Yeah, this is something that Twitter is actually iterating through in terms of really defining the user experience. And that's what it comes down to, not just what the user wants to do, but just the experience that they're looking for. Because what I've found is that most of our service providers were just like, hey, I just want my API server and I want my own Kubernetes cluster.

Coming from an environment where Mesos and Aurora builds infrastructure, where we're seeing the whole data center is one computer to what Kubernetes in terms of just the way that I describe it is a micro orchestration system, where it's just going to be multiple API servers in one data center. Those two different ways of managing your hardware and your infrastructure is where I think the conversion will eventually come into play, where Kubernetes can provide those same services at the level that Mesos and Aurora does.

But the gap still is that Kubernetes is more focused on micro orchestration, micro services. This service wants its own API. It wants its own Kubernetes instance. It wants all of this. But that's where I always go back to understanding what the customer wants, because when I actually speak to customers and they're asking for certain features of functionality, I'm like, all right, the orchestration platform you're looking for, let's not even name it. Let's just understand what you're looking for. I need my instance to run at this interval.

I need my deploys to work this fast. I need my binaries to be published this way. I need to be able to access my TCP ports this way. I need to be able to SSH into instances. I need my logs shipped out. Soon as you start to dive into what the users want in terms of their user experience, then that's when you make a decision on the orchestration platform that we have available today. A lot of people are going to side and go with Kubernetes at first.

Kubernetes will continue to iterate and eventually solve some of the use cases that there may be some voids on the hyperscale end. But in general, my process is always to make sure you collect the right user stories and understand what the users are attempting to do before you choose a technology, because you may make a decision that you'll regret five, 10 years from now, or you have to wait until those features are available with the platform you decide. So that's been more or less, sorry about all this.

That's been more or less the basis of deciding the right platforms to use, especially when you're talking about it at the orchestration level. Yeah. So Bobby, I really appreciate you stopping by and having the conversation. If people want to engage with you, either on the consulting side or the speaker side, how do they best reach out to you? Yeah. The best way to reach out to me is I'm online. Twitter, obviously my profile is BobbyD underscore FL. I'm on LinkedIn.

You can find me under Bobby Doorless. And that's more or less the two avenues that I have available for people to connect with me. Like you mentioned the services I'm really looking forward to expanding is principal engineering consulting service with the focus on SRE at a hyperscale distributed environment. But also I'm doing a lot of community outreach where I'm teaching, mentoring, and coaching the next generation to understand their value and their diverse perspective in this industry. So definitely feel free to reach out.

And again, Keith, thank you for giving me the space to speak to your audience. And I definitely look forward to all the things that you'll be working on in the future. So we went a minute and a half over, but I promise you it'd be worth it. If you want to learn more about the CTO advisor, you can follow me on the web at CTO advisor on Twitter. com is the website. Talk to you next CTO advisor podcast. Bye.