Recovering from Ransomware via All-Flash

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

All right, we are on the roll of talking to my friends from the Tech Field Day community of folks. Last week we had on Chris Evans. This week we have Howard Marks, who's returning to Storage Field Day, but not as a delegate. He's going to be on the other side of the table. Howard, I think this is the one time we can suspend the Howard Marks rule of not throwing things at the presenters. I am going to. The last time I presented it at Storage Field Day, I made it very clear that all of the rules at Tech Field Day with my name on them, except for shut up Howard is always in order, were suspended.

So, you are allowed to throw things. com. It's a really cool series put on by Gestalt IT in which we go and get presented by or presented, yeah, a bunch of tech vendors, enterprise IT tech vendors present to a group of 12 delegates with similar backgrounds to mines. And we go back and forth as we're briefed. I have Howard on because one of the topics that came up when I was at Cloud Field Day 13, the second week of February was backup.

Now, like all things Kubernetes and otherwise, we thought that these were solved problems. And me and Howard were just getting into that. So, Howard, isn't backup a solved problem? I mean, by data domain or something from rubric or cohesity and we're done, right? All of those things help. The problem is multifaceted. I've been doing this a long, long time. And there was a good decade at the beginning of my career where we never said backup, we said tape backup.

And tape and backup were indistinguishable. And tape has problems. You have to find the tape that's got the data you want on it, you got to load it into the drive, all that takes time and effort. So data domain came around and said, look, we're going to deduplicate data to disk. And that's going to make disk cheap enough that you'll buy it for backup. And you can back up to disk. And now when somebody calls the help desk and says, I accidentally deleted the spreadsheet we used to close the quarter, you can go into the console of your favorite backup application, the data mover, and say, restore this file for Joe.

And the file will be restored and be a matter of seconds or minutes. And you can solve the problem with the guy still on the phone. And that was a big improvement. The problem with deduplication and spinning disks is deduplication takes the big backup files that your data mover creates that are in its proprietary format and breaks them up into lots of little chunks that it distributes evenly across the disks so that if two or three backup files contain the same chunk, only has to store that chunk once.

So on the right side, this is a great thing. It saves space. The problem is on the read side. O. on the hard drives on the back end. O. O. as little hard drives. s per terabyte you can do goes down by a factor of 10. Yeah. So this, people don't always grok this. I've learned how to spell grok, by the way, in the past few days. People don't always grok this. If I have a 20 terabyte hard drive versus even, and we actually, we used to talk about this even in the two terabyte hard drive days compared to a six or eight.

This is exponentially a bigger problem. As we put more and more hard, spinning disk in the same amount of space, if we're not increasing the read or write speeds, it's just becoming less and less efficient. We're not, the speed of recovery and reading and writing bits isn't keeping up with the pace of hard drive and storage grok. Well, because the bits are closer together on the disk, the read speed goes up a little, but the move the head to find the next chunk doesn't change.

Right. The mechanical, the mechanics of a drive is still the mechanics of physical mechanics of the drive. And those, you know, and that got pretty well optimized seven or eight years ago, so it's not getting better anymore. So I understand the challenge of dedupe and reconstructing, uh, the bits from that consolidated and optimized perspective. That's more CPU. There's a couple of problems. There's more CPU overhead next level probably today and you can, you can solve that problem. Yeah.

And, but the bigger problem is the random access problem for spinning disk. But I see a lot of vendors talking about instant restore capabilities, the ability to restore on the native appliance. So if I've backed up to my, uh, data domain, rubric, cohesity type appliance, I can restore in place and be running up and running right away. Doesn't that kind of just solve that problem for me? It reveals the same problem. Um, so if you use one of those appliances from those companies you mentioned, um, they're generally hybrid appliances.

There's some flash that they hold their metadata and the most recent data in. And then there's a lot of disk that holds the rest of the data. And if your instant recovery is running from the flash, if you're recovering one VM because one developer screwed it up, they have enough flash that they can promote all the data from the disk to the flash. And in an hour or two, that VM will be running nice and fast. The problem with spinning disks and restores, whether we're talking about instant recovery or actually restoring where you have to move the data back.

And the truth is eventually you have to move the data back because if you're instant recovering on your rubric appliance, you don't have a backup anymore. While, while your primary is running on your backup infrastructure, there is no backup because what used to be backup is primary. Yeah, we'd have this problem disaster recovery. Yes. If you get to the point where you've perfected your disaster recovery and you can move over at some point, you have to move back and be honest, you know, I come from the, you know, my background tier, tier one, tier zero storage arrays, supporting SAP and other transaction apps.

My backup infrastructure, while I can limp along, you know, I'm not going to, I'm going to get some calls about SAP interface performance and curing time. You know, the backup infrastructure is exactly that, the backup infrastructure. Yeah. And instant recovery is a great thing because it gets your users back up sooner. Right. Generally, they're going to be slower until you move them back to the primary. Yeah. Unless you're just a small environment, then, you know, you still have the problem that you got to get back to production.

Yeah. The problem is when we start talking about things like ransomware, where we're not talking about, I got one corrupt VM, I got one database where a developer did something stupid and I got to restore that from a snapshot. I got thousands, maybe hundreds of thousands of encrypted files before I know that the ransomware attack is in progress because storage systems haven't gotten smart enough yet. And we're working on some things about that to detect the attack in progress. Ideally, a storage system will be able to go, this user normally stores four files a day and he just wrote his hundredth file today.

That's suspicious. Maybe shut him down for a minute and call the administrator and see what happens. Storage systems don't do that yet. So you run a backup Friday night and over the weekend, your backup software runs some analysis that goes, oh my God, the change rate went through the roof. There must be a ransomware attack going on. And Monday morning, you come in and you see the report. But the ransomware has had all weekend to encrypt your files. So when you go to restore, it's not gigabytes you're restoring, it's terabytes or petabytes that you're restoring.

And now pure restore performance becomes important. It's been a truism as long as I've been in the business, you only back up to restore. Nobody backs up for the sake of backing up. You back up in case you might restore. And we're reaching the point where the scale of restores is changing and the performance asymmetry of the existing systems is in exactly the wrong direction, where they're fast to back up because that was the decision, how people chose those systems five or six years ago.

It's like, well, I have a four hour backup window. I have to move two petabytes in four hours, therefore I need something that moves data fast enough that I can make my commitment. But when the restore is a tenth of that fast, now you've got vice presidents looking over your shoulder going, when's this going to be done? And I always hated that. Yeah. So this is kind of bringing up a couple of stories that we've had in the back channel.

I'll put a link to really, really fun video from Linus Tech Tips, who built his own petabyte scale NAS. I'm not saying you shouldn't build your own petabyte scale NAS, but this video is a great example of why you probably should consider not building your own petabyte. I have one thing to say about that video. It was great. It was entertaining, right? If you build your own storage array, you still need to check it every week to see if any disk drives failed.

It is. Building their own storage array was their first mistake. Letting multiple disk drives fail without being replaced and sit there for weeks on end was a bigger mistake. You know, when you have, again, when you operate a data center, these are some things, you know, I'm not going to say there's not a place for building your own storage array. HPC, it happens all the time. There's- Oh, wait, I work for VAST. So, disclaimer, Howard Marks works for VAST data that deals with petabyte, QLC, and Optane quality storage.

So, you know, there's some bias. And our big line into HPC centers is just, you know, you could build your own, but you need three guys for that. You know, I would say you need three guys for that. Or you could buy from us and put those three guys to use doing something useful. And that is the point, is that you end up in a large data center with so much spinning rust. There's literally somebody's job to go through probably once a day to once a week and just look to see how many drives has failed.

And they have a cart of drives and they're just replacing drives. This is kind of, when you, especially when you get to the petabyte scale of spinning disks, you're going to be replacing drives constantly. That 1% to 2% failure rate becomes material when you have 1,000 drives, you know. Or 10,000 drives. Or 10,000 drives. You're going to be replacing a lot of drives. So, Howard, let's- You just reminded me of one of my college professors, got his PhD at the University of Pennsylvania, pushing the cart down the aisle of ENIAC and changing the tubes.

So, we haven't come that far. We're just changing disk drives instead of tubes. They last years instead of weeks. Yeah, that is, you just aged yourself. Or your professor lived for 100 years. So either one is, someone's old. I'm old, that's true. I did used to have a podcast called Gray Beards on Storage. Yeah, I think I know something about that podcast. But getting back to, okay, I think I understand the problem, is the appliances are great parts of the strategy.

When you have petabyte scale backups to backup, and you need to eventually get that back one day, how do, and I guess we'll go into a little bit of your bias, because you work for Vast, and you guys are kind of looking at solving this problem. How do you- Well, just like all flash has become the default choice for primary storage. If you've got something that needs to go fast, you go all flash. We've reached the point where, with the efficiencies that we at Vast build into the system, we can deliver a system where flash works for backup at a price you can afford for backup.

And now you can have that instant restore and have the users on the instant restore running fast enough you can have, if your ERP system goes down, you don't have to instant restore it and then say, I'm going to allow four users in to do the emergency stuff until tonight, we're going to storage vMotion that back to primary storage, and it's going to be fast. You can be fast right away and get right back to it. If you need to restore a few thousand or a few million files because they've been encrypted, you're reading them at tens or hundreds of gigabytes per second.

Our performance asymmetry is exactly the opposite of a data domain. We are slower writing than we are reading. So if you made your backup window, you're going to restore faster than that. You're probably going to restore as fast as the system we're restoring to can take it. So I think that's a pretty honest tradeoff and talk about the tradeoff. If you need to restore and restoration is important, then flash makes sense. If you need to write it, if there is a tremendous amount of change throughout the day and you need to write and write, performance matters and you need both.

You're kind of you know, I don't know if there's a solution if you just need both. That's a really interesting point. Our write performance is slower than our read performance. But for a comparable capacity, we're still faster than the data domain on writes. So I then I know a few of my data center centric folk because we talk to cloud folk as well. They're wondering like, wow, people still buy data domain. Not a slight towards the data domain. It's just that the rubrics and the cohesities of the world have really been done a really great job of of making an argument against data domain.

So this is a data domain. But generically, we're speaking about this whole appliance market. Well, yeah. And we're talking about appliances with spinning desks. Yes. And the same things apply. You know, now rubric and cohesity have more flash than data domain does. So it's not as big a problem. But they're also not terribly well designed for petabyte scale. So we'll save the conversation about cost for another podcast, because there's obvious question. If I'm going to replace all spinning rust with flash specifically for backup, I don't do that because of the expense associated with flash.

But that is more that goes more to the marketing message of VAST. And we'll save that for another day. Howard, how will people find out more about you in your day job at VAST? Well, me, the best place is Twitter. I'm at Deep Storage Net, all one word. com. I'm happy to talk to anybody about VAST at any time. That's what I do. I'm the mouthpiece. I'm the explainer. Here's how our stuff works. And, you know, basically the whole idea of VAST is to use cheap flash.

Treat it really kindly and bring the cost down to the point where you can replace all those disk based applications with flash and not have to wait anymore. No one likes waiting. No, I don't like waiting. com or follow me on the web at CTO Advisor on Twitter. I am a prolific tweeter. I send out a lot of stuff, including videos throughout the week and content. That's it for this episode of the CTO Advisor. Talk to you next CTO Advisor podcast.