The Return of Greg Ferro - Programmable ASIC - CTO Advisor 087

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

All right, welcome to episode 87 of the CTO Advisor Podcast. You know, I don't know if we're 86, 87, 88. I don't know if it really matters. I'm no longer a geek, so I don't have to be exact in my maths. But packets can come out of order. Yeah, there you go. Packets can come out of order. Speaking of packets can come out of order, we're joined by Greg Farrell. Greg, it's been a few months since we've had you on the podcast.

And it's good to be back. Thank you so much for inviting me. As always, it's good to talk to you, Keith. And as always, I'm in between you and beer 30. It's early in the AM here in Chicago, but it is the end of day Friday for you. And somehow, I think you have a nice room temperature stout waiting on you somewhere. I certainly do. So my goal, I take one day off a week, Saturday. And the rest of the week is spent doing my thing.

But Friday evening, I wrap up usually early in the afternoon and then head off to the pub, where I'll proceed to have a few quiet beers and reflect on the joys of being a human inside of the technology sphere. Right before we hit record, you gave me a number that I thought was extremely interesting and relevant to this week. This week just happens to be Cisco Live US, extremely large networking event held in Orlando, the home of Mickey Mouse. And the team there was really excited about announcing DevNet and the number of developers, quote unquote, developers on DevNet.

And that number was 500,000 registered developers. net in which you get networking professionals to come sign up and do personal development and learning about industry-wide things that's important to developers. What was that number again? How many folks do you have registered in that group? Well, our membership system's open to anybody as a free membership. And we have about 5,000 people on that today. And then across the entire PacketPushers network, between the podcasts, the newsletters, we think we've got somewhere between 50,000 to 75,000 unique people that we engage with every week.

So I find that amazing that a company that produces, how many shows do you guys produce a week? We have five channels that we produce. And generally, we produce three or four a week. So one of the things that I pushed back on on Twitter with the Cisco number, 500,000 developers, I thought that number was kind of curious. Because if you have 500,000 developers that are active in anything, we would see some amazing change in the industry. Like that number, while maybe valid, didn't pass a level, a sniff test to me.

Like, OK, 500,000 developers, there should be some amazing products and automation. There should be some GitHub projects that are super active. And the real number turns out to be around 70,000. The Cube X, the DevNet folks, how many of those people were active? And it was around 70,000 per month. Yeah, so you're looking at 500,000 total registration, 70,000 MAUs. It's a bit like Facebook talks about, I think, whether they've got like 4 billion people registered and a billion monthly active users.

So that number is about right. I think that a lot of people who use DevNet don't actually use it for developer. They use it for hands-on. So if you're trying to get up to speed, if you're a partner, a Cisco partner, you'll have access to various labs and testing environments where you're just using it to learn or train or to do some self-education. So I think it's a little bit specious to call them developers. But, you know, companies need to have good metrics to make shareholders happy.

Big companies like Cisco needs to say that it's under a lot of pressure from Wall Street to prove that it is a software company, that it is a cloud company, that it looks like Google and Amazon and Netflix, you know, and that sort of recurring revenue model. So it's much easier to say developers are on developer net and get away with that. So that's, I think, from a senior architect or CTO. It can be frustrating when you look at a number like that.

And then even with the 70,000 actives a month would still be a very impressive number. But I suspect the same as you that that 70,000 aren't 70,000 people coding. And when we look at the, you know, everyone is telling us, well, network people need to become coders. Network people need to become coders. And Cisco publishes this number and you think to yourself, man, am I that, I know I work in the enterprise and I'm always behind the curve, but am I actually that far behind the curve?

I don't think so. I don't, my view on this is fairly clear. I think everybody needs to learn how to script or to program because you need to know how that works so that the day comes when you're operating a piece of software, you understand what's happening inside it. So my knowledge of programming has informed how I use everyday programs like Word and email, my ability to create filters, to make my mail come under control and things like that. And, you know, if you do start to run some of the software defined platforms, be it software defined storage, software defined compute, software defined networking, you know, look at what's happening with hyperconverged.

There is a good chance that you could write a script that actually would be incredibly useful to your business. It wouldn't take you more than a day or two and you could do a certain thing. And so in that sense, yes, you're probably going to need some programming skills. It'll make you a better user of software. And it also lets you be able to do a specific thing that might just be uniquely useful. I do not believe that you are going to be a developer in the sense that I'm going to sit down, map out an app and then start coding for six months and boom, there's my app ready to go.

So there's a gap between doing something useful, like a little tiny microservice that does a thing and a big difference between building an entire app that is unique to your organization. So that brings me to another topic that I was listening to Network Break this week that you normally record on Fridays and then you publish the Monday, Tuesday, something like that. Monday afternoon. Monday afternoon. And I'm like full steam ahead with this concept of, I think the chip set is Telfino?

Yeah, Belford Telfino. Belford Telfino. I absolutely love the concept. And for my audience, I think it's probably best if you explain the platform and then I'll explain why I'm like super excited about it. I'll give it a go. So in networking switches and routers, the way that they work today is that a packet comes in and it goes into, when a packet goes into a router, it goes into the router and then it goes into the router when a packet is received or a frame is Ethernet frame or an IP packet is received, it comes off the physical interface and then it's handed off to a bunch of chips inside of that device.

And then it gets forwarded through and they get it sent back out another port. Those Silicon chips that are inside there, there's a heart of it is an ASIC, which is a dedicated piece of Silicon and it's normally fixed. That is the packet comes in, you can do this, this, and this to it. You can do a cost tag, inject an MPS tag. You can do some NAT, rewrite the TCP header or IP header, but that's really about it. You don't get a lot of choices.

The functions are set in the factory when the ASIC is made. What the Barefoot Networks have done is they've given some thought to how you design a chip. And then they've actually designed a chip, which says, why can't I just write an ASIC that just is able to rewrite any part of the packet as it moves through the device. And in the old days, two years ago to old days, it was like you had to build an ASIC. 6 nanosecond gap before the next one starts to arrive.

So networking is very different from computing because we have to do things in real time. And the packet has to clear the switch as quickly as possible. Otherwise you're injecting latency. The more latency you get, the impact of latency is actually to the square. It's not like, oh, if I add 10% latency in the network, it's 10% slower. It's not, 10% latency actually adds up to 100% slower transactions between servers and client. So latency is a key consideration. Usually you don't know this because the network is so fast that most people don't notice.

And to be honest, most people aren't doing stuff which is real time or high speed, right? So you don't notice, but that's the fact of life. So the challenge here with Tofino is when the packet goes in, the internal architecture of the Tofino ASIC lets you just rewrite anything. And you can do it over and over and over. So I can bring it in. I can take the VXLAN frame coming in. I can strip the VXLAN header, then look at the internal ethernet header.

I can strip the ethernet frame and look at the IP header. I can strip the IP header and then put a new IP header, a new ethernet frame, a new VXLAN. I can route it. I can switch it. I can NAT inside of the VXLAN. And because I'm not holding state in the ASIC the way that the legacy ASICs do, I can have these vast numbers of VXLAN tunnels, things. And I can just program it. To take this a step further, that's exciting enough in its own right.

But to take it a step further, what the Barefoot crew did is they invented a language called P4. They have a funny word about PPP, programmable, something, something, something. And this language, it basically lets you do anything to an IP packet or an ethernet frame, anything at all. And you can rewrite it. So for now, let me give you a classic example of why you'd be interested. And the current use case that they're selling it heavily around is this thing called networking and telemetry.

So what you're able to do is as a packet comes through the Tofino chipset, I can put a tag in there that I make. I write a P4 program and I can write my own tag. So instead of having to wait for the ITF to define a tag for telemetry or a standardized format, I can just make my own, which is, and then I can track the tag across the network and boom, I've got a telemetry system. So when you, I get excited whenever I hear the term telemetry.

I think of data that I can feed into a orchestrator. And then that orchestrator can make decisions based on that telemetry data. So for the networking, one of the things that I'm really struggling with, like conceptually, or at least when I look out like at the Kubernetes program and market is that, I have this ability to do multi-cloud across different service providers because they all support Kubernetes. And I'm starting to get to a point where I have a standard control plane across workload orchestration.

However, one of the inputs into workload orchestration is network latency, et cetera. So I know this is a little bit higher up the stack than what Telfino works at, but the concepts go all the way down. They scale to scale down to data center. So- Well, what they're doing with P4 is they're now using P4 in CNI, in Kubernetes CNI. So the idea is that you could use P4 to define how you move through the eBPF interface in Linux. So instead of actually saying, I wanna, and writing a C language processes to say, this is what I want to do with this packet.

I can actually now use P4 as a scripting language for packet munging. So we interviewed a couple of folks while we're at ONUG, I think, the open network user group a couple of months ago. And we were talking about CNI and the challenges of doing NFV, raising network functions virtualization, but in containers. So the containerization of that process, I'll put the interview in a show notes. It's amazing the work those guys are doing. It's a great roadmap for just saying, I'm going to containerize any application and the pains you have to go through to modernize the application.

NFV is a great example of that. And Telfino is one of these things that kind of enables that or enhances, because we still have the problem of legacy network gear that we have to integrate with that doesn't have a Telfino trip in. But to kind of drive home this point, I think I can loosely, I think it's fair to loosely compare what they've done, what Barefoot has done with Telfino to FPGA in a traditional server. Yeah, to some extent. FPGA is a field programmable gate arrays, and that means that you can actually load programs into them, and then the programs will actually run in the ASIC.

That is fine as a concept, but it's not the reality. Because of the way FPGAs are, they're general purpose CPUs, and they're like programmable silicon in its own right. But they're still much too slow for normal networking use. And I think that lends to the question I was going to ask you to kind of tease out for me. Because when I look at something like DPDK, I don't know if it's even named that anymore, but Intel DPDK, and putting it with FPGAs, Intel publishes hero numbers.

For those of us that's not networking experts, help us tease away like the difference in scale. Like, okay, DPDK with, enabled with, accelerated with FPGAs are able to get this magnitude of performance. You know, scales of magnitude is great enough for me. Versus something like Telfino, which is purpose driven to a lower level, specifically for networking, allows you to do this scale or these types of applications versus general purpose CPUs. So usually you're talking about nanoseconds versus hundreds of nanoseconds. Okay.

So DPDK is an acceleration engine that works on Intel's x86 platform. So what they've done is put network acceleration capabilities into the silicon of the CPU. And the DPDK is a set of SDKs, like software APIs, that lets you offload the processing of that onto dedicated silicon. In a sense, it's kind of like, what we're seeing is the emergence, especially people doing high performance compute and cloud, they use dedicated network adapters that have high capacity, high performance on them. So NoviFlow, Mellanox, things like that.

And the Mellanox NIC has, there's one Mellanox NIC which has 256 ARM cores, as well as a fully programmed ASIC. And you can use either or, right? So these things can drive 100 gigs line rate all day and no CPU burn. Whereas normally, if you're driving 100 gigs out of an x86, you may be losing somewhere from five to 15%. You could, or you could actually see before DPDK came around, we were seeing people lose an entire CPU thread, a normal CPU die would have four cores, you're losing one entire core to talk at 40 gigabits per second.

With DPDK and the silicon that goes inside of the chipset, now they're using a combination of what's inside the CPU, as well as the chipsets on the motherboard, but for all intents and purposes, the CPU. DPDK allows you to accelerate that to 100 gigabits per second, and burning five to 10% of one CPU core. If you go all the way out to the extreme, we use dedicated NICs, you're actually able to do line rate with zero hit to the CPU, or effectively zero hit to the CPU, because what you're actually doing is offloading all that processing away from the CPU onto a NIC.

And with the price of service today, and the price of memory today, that is highly desirable. So bringing this up to, there's two parts of this that I want to go with this. One, the programmability, and then two, the efficiency, and then let's say a third piece, which is the true piece of it, the efficiency gained, and what does this mean to all these cloudy folks out there? So there's economies out there, but I think the first thing that, so if I have a traditional data center, and I built my data center around Telfino, I think you can probably describe this much better to me.

Practically, what does that mean for my operations now that I've built around Telfino? So what that means is you've now got a switch which can do any functions that you need, whether it's MPLS, whether it's VLAN tagging, whether it's QNQ, whether it's VXLAN overlays, whether you want to, whatever the innovations come up in the future around VXLAN overlays, we can do those. The ASIC is programmable, it can do all of those. The price to be paid is it's a little bit slower than a normal switch.

If you're a mid-sized corporation, let's say you've got 10 racks, you'll never know, you won't care. But if you're a cloud, you might know. The flexibility then comes from the software that sits on the switch itself. So whether you buy your Telfino in a white box, or whether you buy a Cisco Nexus 3400, or whether you buy an Arista 7170, the point is now you've also got the capability to put P4 programs on it. Now, you might not put a P4 program on it, but somebody's SDN platform might.

Some telemetry platform might want to put a P4 app into the switch so that you can say, on this server, so let me paint you a picture here, try and paint a verbal picture. Let's say you've got a performance problem, you've got an ECMP spine, you've got six switches across going down to your hyper-converged stacks, right? And then you've got four ECMP spine switches at the top. So you've probably got like 240, 400, 10 gig ports, 25 gig ports. And somewhere along the line, you've got this intermittent problem.

And the most common cause of this is that in an ECMP network, everybody's got an equal path, but it could be four, eight, 12, 16 paths from a destination. So if just one of those paths is not working, how do you know? Right? In a multi-path network, how do you know if one of those paths isn't working? I have no idea. I don't even know if, I can't think of a tool that'll help me determine that problem. Now, if you've got a physical problem, that is your SFP's failing or the optical cable or the coax cable or whatever it is is failing, you might get some physical counters off the devices themselves because they monitor the PHY, the physical interface.

But if you've got some other problem, it's possible for things to have failures that don't get detected further up the stack. And I want to tease that out because people, if you're not a networking person, you may not be able to really fully comprehend this. So I had this crazy problem with multicast a couple of years ago that we couldn't find, I mean, all the counters, everything looked perfect. And what ended up being the case is that from the core to the multicast participants, the switch is actually receiving the data.

Once the data was transferred to the edge and the closets, and these, we were using 6509s as closet switches. I mean, these are closet switches. So we were thinking that, okay, the 6509s at the edge were overkill for what we were doing. So we never even thought to look at that as the problem. But in reality, the problem was, is that when the bit rate of the video that we're sending was too high, the 6509 CPU couldn't keep up with the request to send the data across all the ports on the 6509s.

That's right. So what actually happened is- That didn't hit the counter. Yeah. What actually happens in that situation, this is very specific to multicast. What a lot of people don't realize that in a switch, it's not the CPU that does it. There's actually a special chip in there, special chip function in there. And the early versions of multicast were never designed to operate wire speed. And they were only ever designed to work for a fraction of the capacity of the switch.

So when the frame comes in, and let's say you've got to send it out 40 ports, you've got to duplicate that packet as it moves through the silicon at line rate. And we're talking nanoseconds here, and then send it out all of the other ports. And what happens is that little chip that does the duplication actually dies. It's just not rated for high speed multicast. And it's also not logged. So that is an example of a problem in the picture you've painted that there could be a modern day equivalent to that in that whole design.

So how do you troubleshoot for that? So the way that you do this is you use your P4, you write a tag that goes onto it. Maybe if you're lucky, you've got an edge, maybe you've got a CNI or a VM, and you can use a P4 program to inject the tag. Maybe it's a 64 bit tag or something like that, whatever size makes it happen. In that tag, you can write the P4 app to say put the server name in there.

It came in on this switch, and it came out on that port, or received on this port, went out that port. And then every time you move through the network, you write that tag into the IP packet. Make sense? So it comes out of the server, it's got a tag saying I came from this server, I left on ethernet one, and here's a timestamp. And then when you get to the switch, it receives it, and then the P4 program on the switch says, here's another, I received it on port three slash 22, I arrived at this time sync, and I'm now going to dispatch it out that port.

And so you can now, and then when you get to the server at the other end, you get a, you know, so forth. What you can then do is use either the network or the servers to stream that telemetry away to a system who can then analyze that and be able to say, hang on, I can see the packet leaving the server, I can see it crossing the top of rack switch, I can see it hitting the ECMP spine, and then I get a 10% packet loss from that point on.

Therefore I know, and what you'll probably see with the right telemetry system or the right visibility is you'll see that there's one interface on the switch, which is dropping just enough packets to drop 10% of packets in a given string. So that's a great example of like how operationally, well, that's one operational example where P4 and programmable ASICs come into play in operations. Another one is like, let's say one, you can buy that software in theory from your vendor or a third party.

Another scenario is that like a practical problem that if you've run a network, you've come into the solution, you've just spent 50,000, 100,000 on this new whiz bang switch that's been out, let's say for a year, not even a year. And there's a new feature. And it's some new, there is some feature that requires a tagging of a packet. So it's in the data path. So VXLAN, the classic one is VXLAN is pretty much the dominant overlay network. Right. But you might still wanna use NVGRE or NV03, or maybe you actually wanna create your own, maybe you wanna use some MPLS, or maybe you actually wanna create your own overlay protocol.

And you go to your vendor, you say, you know what? I just bought this switch yesterday. It's been out for a year. VXLAN, I wanna add VXLAN to this new $100,000 switch. And the answer is... Most often buy a new one. Exactly. So there is some programmability in the current generations of chips, like the Broadcom Trident 3 or the Jericho or the Dune. Those chips have some limited programmability. So this is not as much of a problem as it was a few years ago, because companies like Barefoot have actually promoted the idea of programmability.

So in the latest Trident 3, there's a three-stage programmable step in and a two-stage programmable step out. But five programmable steps is quite limited. When you think of that you're coming out of it, if you've got an overlay network coming in, so you're receiving a frame, it's got the outer Ethernet header, then it's got a VXLAN header, and then it's got an Ethernet header inside it. So now what you wanna do is strip the Ethernet header off, strip the VXLAN header off.

Maybe you wanna NAT that, then you wanna route that. So you've got to rewrite the header for NAT, rewrite the header for routing, because you've got to change the TCP counter, decrement the counter. And then you wanna re-encapsulate it back in a VXLAN, that's another one. And then you've got to put the Ethernet frame back on the outside. So that's four steps. Then all of a sudden, normal low-cost silicon that comes from Broadcom can't cope with that because it's dead.

I love getting into these deep conversations with you. We've run out of time. I hope we didn't go too deep for the audience, but you know, we're talking to technical folks. I think we kept it high enough that this should play well. I love this. I'm like super excited for Telfino. I love that. It's an interesting idea. I think the real magic here is the software-defined networking apps. The trick here is that P4 is an app that runs on the Switch.

And people have to start thinking about operating systems on a Switch as an operating system. And then on top of that, there's a whole bunch of apps. And some, maybe you're running the BGP EVPN app, maybe you're running an OSPF app, maybe you're running an SNMP app or whatever it is, but there's also going to be apps, whether they run in a container on the Switch or whether they run native in the Linux kernel that runs on the NOS, that's going to be able to do things like use P4 to program in the forwarding plane.

But we're also going to be able to use P4 on servers, in virtual Switches, in container networking interfaces, in other places as well. So we can actually have a universal language for packet managing, whether you're on a hardware or on any other virtual platform forever. That's the unique part about P4. And with networking, it's the disaggregation. So we've unbundled the Switch. So now the Switch is like an x86 server. Then we have an operating system, which is a version of Linux that works on the hardware that's unique to networking.

And then we have a bunch of apps that sit on the top. And then over that, we have an SDN controller that can talk to those apps and coordinate. Networking is uniquely a distributed problem, distributed computing problem. You don't have one Switch, you have 20 of them or whatever the number, you know, routers and files, and they all have to come together into a coherent system to work correctly. And that's what we haven't done. Even though we run distributed systems today, we don't run them as a single unit.

We actually treat them all as unique little unicorns in their own right. And the SDN is really turning this around in the same way that a hyperconverged changed the way you look at your VMs. And then that brings us, and I think that deep dive into networking as a subsystem within your entire distributed data center becomes a little bit clearer. And as you paint the picture, you know, now you're starting to talk to the higher level functions of, in theory, and the market will bear this out, but I think the direction it's headed is that the SDN controller then talks to Kubernetes and provides telemetry data to Kubernetes to be able to make decisions on weather, place workloads, et cetera, et cetera.

But this is the deep dive into understanding how, you know, the movements that needs to happen so that all this agility that we're trying to build at this cloudy layer, this orchestration, this workload orchestration layer, everything that needs to take place underneath to be able to guarantee or at least communicate performance information up to that higher level stack. All this extraction that we're doing at the Kubernetes layer requires a pretty solid understanding of how these subsystems work. I think that's an interesting thing about Kubernetes is you're not going to be hand carving your artisanal network like you do today.

Right. Because Kubernetes, even above Kubernetes, there's something else, right? There's still an orchestration system above Kubernetes. So you're going to have Kubernetes, a container which has its own container network interface, which probably will be a plugin that then talks out of a VM. And there's good reasons to put containers inside of VMs. And then the VM is going to have a virtual switch and that's going to, and you might have 200 VMs in a single server, say each one of those VMs is running 200 containers, and then you get to the NIC and then you plug into the virtual switch.

So. Yeah, and there's a whole, I think another conversation that one day we'll have you on to talk about, which is that orchestration, because right now we're just looking at Kubernetes as, and that Kubernetes solves a specific problem. When you're talking about all of orchestration and that Kubernetes is laser focused. What happens when you put a container in Google Cloud, like Cisco, of course, it's just come off Cisco. Well, Cisco Live, as we started out at the conversation, they're talking about doing a partnership with Google Cloud.

So now if you've got a container in Google Cloud, how does it talk to a container in your private cloud? Exactly. In theory, you know, and then Netflix has pretty much worked this out with their private, with their Titus project that they just open sourced. But they do massive, you know, I think it's that I think they is somewhere in a realm of last time I looked, 200,000 hosts a day in AWS. They still, people don't realize this. Netflix still has a pretty sizable private cloud.

And these containers are orchestrated between private and public cloud. And they chose to build their own and then also leverage Mesosphere to do higher level orchestration. So there's a lot of kind of things that we haven't teased out with, you know, kind of big media and general public on the challenges associated with doing true multi-cloud Kubernetes orchestration. Well, the biggest part of Netflix's infrastructure, this is just an aside, we shouldn't probably go down this right hole. The biggest part of Netflix's computing infrastructure is actually not even on their own premises, not in Amazon and not their own.

It's actually at the edge of the network in universities and telco pops, where they actually have CDN nodes and they have thousands of those, way more than they have in AWS. Yeah, that's a whole, talking about the edge, because there's no way for them to provide the service that they provide at the quality that they do without having a tremendously big presence near, closer to the edge. There's no point in streaming, you know, from the nearest AWS node, when you've got, you know, let's say you're in a town with a hundred thousand people, let's say 5,000 people are streaming Netflix at seven o'clock at night.

There's no way you're going to stream that from somewhere, you know, 10,000 miles away. There's not enough bandwidth in the world to make that happen. So you have to find ways to bypass the bandwidth for some things. Yeah, I've had to go through the pain of doing video at a larger scale for an enterprise. I was doing 13,000 nodes across a hundred sites and it is a fun challenge, but it is a challenge. And these guys are doing it at a edge, at least I control the edge.

They don't even control the edge. The last mile of delivery is not even in their management. I look towards them to see directionally where the enterprise will eventually go, because I think the enterprise will follow, of all the hyperscalers, I think the enterprise will follow Netflix's example most. And then, you know, like the, obviously enterprises won't have the challenges that AWS, Google, and Microsoft have. I think Netflix is probably one of the closest examples to a real enterprise app. Yeah, I suspect they're closer to an enterprise than what, you know, the kids of Silicon Valley are doing.

You know, as I've said to you before, when we've had conversations is a lot of what AWS does is actually optimize for companies that aren't expected to succeed. What I mean by that is the great part about the cloud is let's say you're a Silicon Valley investor and you're dropping a million dollars on a startup on our idea. And if you get three months in and you realize it's a dumb idea, what's the one thing that you're optimizing for? Failure. Failure.

And so let's say you've got a portfolio with a quarter of a million, you know, $250 million in it. You're going to invest in 250 startups and you're expecting 245 of them to fail. What do you want your portfolio to look like? You don't want to have 250 leases. You don't want to have chairs and desks. You want to have the bare minimum that you can have so that you can just walk away from as many of these as possible when the ideas don't work out.

And hence, AWS is good for them. So, Greg, we're a little bit over. We can probably go for at least another half an hour. I gotta get you on more often. I love these. You really challenged a lot of my assumptions and get me to thinking. I hope the audience has enjoyed it. For those who have not followed you, I don't know why not, where can they find you on the webs? net where we've got all of our podcasts and blog posts I publish fairly regularly there.

And I expostulate on a regular basis with a range of inspirational cynicisms as at Ethereal Mind. All right, and of course, you can find me on the web. com is the website and at CTO Advisor is the handle on Twitter. We'll talk to you next CTO Advisor podcast.