A Primer on Ransomware with Melissa Palmer

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

Hey, how's it going? It's Keith Townsend. You're listening to another episode of the CTO advisor podcast. We have a new guest. Social media is funny. I've known Melissa Palmer for over 10 years now via the Twitters. I actually know her as vmiss33. It's hard to, you know, Melissa is one of those things. It's hard to call somebody by their real name because you're like, oh, Twitter handle. Go to the conference. Exactly. Have you seen vmiss?

Everyone knows who you're talking about. net. net. No 33 on the URL. Somebody else took vmiss on Twitter, so I had to add the 33 on the end. net. Great information around ransomware, ransomware protection, recovery. And that is the topic of today's episode. So, Melissa, first off, welcome to the show. Keith, thank you so much for having me like we've known each other for so long, but I haven't been here yet. So I'm excited to be here.

We're excited to have you. So a few weeks ago, I teamed with another analyst firm to do a briefing on data protection around ransomware. And I got the big Jesus scared out of me. It's scary. It is. It is super scary. So we're going to walk through the basics and then kind of the next steps for people when it comes to ransomware protection and not just protection. I think of any point in history now, you absolutely must have a recovery strategy.

Absolutely. Because it's you can have the best hygiene possible and still get hit with ransomware. It's something that I worry about, you know, pretty seriously now from our own environment. So first off, what is ransomware? Ransomware? Well, it's going to hold you ransom. Right. So it is the evolution of all the things we're used to. I like to cite the example of the Melissa virus back in the 90s because my name is Melissa. Right. That was like the first big computer virus.

Right. So it's kind of the evolution of the virus malware chain. The whole gimmick is they get in. They do something to you. They either encrypt you or steal your data or both. Right. And then they hold it ransom. They want you to pay them so that they'll decrypt your data or they won't release whatever pictures they found on somebody's shared drive or something like that. Right. Or, you know, I like to use that example. Usually it's financial data, customer data, all that kind of stuff.

Right. They want to extort you into paying the ransom. So ransomware is the maturation of a worm like Melissa. When Melissa came out in the 90s, I thought, wow, this could have been really, really bad. It was already bad. And now it's worse. Yeah, it is way worse. And this is way worse than I even imagined with Melissa, because I think, man, you know what? They could have they could have just deleted all of my data. And that would have been, you know, unfortunate, unfortunate.

But back up, you know, back up and restore, you know, whatever the big vendor of the day, you know, whether it was CA, Commvault, whatever, I could have, you know, we had pretty good backup hygiene. So let's go kind of to that next stage of questioning, which is how do I get ransomware? Like what's the common way of entry of ransomware into my environment? So here is the scary part, and which is why I don't actually like the term ransomware prevention. And there's a lot of research studies out there.

So folks did one. Basically, the number one way the hackers get in is phishing. Someone clicked the link they shouldn't have. Right. And then they're in because they got credentials, they got whatever. So that's really, really disturbing. We always have a huge human element when it comes to security. And until we remove all the humans, we're never going to be able to be 100 percent secure. Right. So the barrier to entry is a matter of a phishing attack.

And they're in. Right. And here's here's the scariest part of this. Right. It could be pretty much any generic hacker. Maybe they don't have a lot of skills, but they figured enough out. And then there's something called ransomware as a service. So most of these ransomware groups operate in a model where they pay affiliates of theirs a certain portion of the ransom. So if I'm a generic hacker and I can get into your network, I can then get everything I need from the ransomware gang, from detailed instructions on once I'm in, how to compromise things, how to destroy things, what data to get and how to run their encryption tools.

And then they take over the rest. I'll get I don't remember what any of the percentages off the top of my head. I'll get a percentage for bringing them in and doing the deployment. And then they'll go in and they will do all the negotiations, take the payment, et cetera, et cetera. So the barrier to entry for ransomware is actually very, very low. So let me get this right. So it is as simple as. I think a serious scenario, you know, we we shouldn't give our corporate laptops to anyone to do anything.

No, never, never. We never do that. We never let our significant other boyfriend, girlfriend, associate check their Gmail. Oh, please. No, please. No, we never. We never let that happen. But let's say that we did. Yeah. If I as an attempt can help have that person get into our corporate network or even something as simple, something as nefarious as someone specifically targeting someone saying, wow, I could get I'm I'm loose. I'm going after that CTO advisor.

I'm loosely associated with Keith and I can use his laptop. He's so generous with his laptop and his Wi-Fi. When I get home, I can jump onto his corporate network because I know that he's permanently connected to the VPN. And I'm into his data center and he'll never know how he was infected. And I get a V, I get I get a piece of that ransomware. Yeah. And that's the scary part, too, is a lot of people never figure out how they got it necessarily.

Right. So that's when there's a lot more. There's a lot more to it. Right. A lot of times organizations will involve an incident response firm when something happens to come in and figure all that stuff out for them because they can't do it themselves. Other times they'll pay the ransom themselves. Other times they'll just say, ha ha, and restore for backup. It really depends. But it really is a very sophisticated kind of business model that enables us all.

And I think a lot of people miss that piece. They just think, oh, all the hackers won't get in. Well, if you're an organization, right, any hacker could be scanning the Internet for your Internet connected servers, devices, because, you know, V centers are on the Internet. It's disturbing. I don't even want to talk about that. I don't know how many times I have to tweet this. Don't put your V center on the Internet. Please. Workspace one is such a mess right now that we actually took our Workspace one, our Horizon instance offline because we couldn't keep it secure.

We just we don't have the internal skill to keep it secure. So we feel back to a different remote access methodology because it just the risk was not worth the value. And this was a direct response for me getting scared about ransomware. And we have pretty good backup hygiene. We have a really well-known solution for backup of virtualized environments. It's been working really well for us. And. I kind of figured pre kind of my education to ransomware that that was enough.

I just recover from backup when, you know, I blew up my DNS a few weeks ago and it was a horrible time for us. But we had backups and we just simply restored our environment, our entire data center from backup. Why isn't that good enough for ransomware? Well, the fact is, most people haven't actually tested doing it, and it could be good enough for ransomware. If you can go restore your whole data center and meet your recovery time objectives and meet your recovery point objectives.

Sure. Go for it. But until people have actually tested this stuff, verify that they are taking good backups. I mean, you know what you're doing. Right. And verify that they can recover in the period of time they need to to stay in business. It kind of doesn't mean anything. Right. Unless you've tested that recovery plan. How do you know you can actually recover? And a lot of people don't test it. That's the issue. Or they haven't backed everything up.

And that's the issue, because once the ransomware is in there, that's it. You're done. So if you've never tested before and you found out, oh, I guess what? I forgot to back up all those critical systems. You're in trouble. You really are. Yeah. The one of the things that kind of scared me was the idea that they could actually encrypt. They could actually target my backup system. I don't do offsite backups because I don't have a business case for it.

Meaning I've I've accepted the risk that if I had a smoking. Whole event and my data center was gone that I'd have to spend the next two to three weeks rebuilding my. And that's fine for my business. But you said that you said the exact thing. You've accepted the risk. You've looked at the risk. A lot of people don't. Or they look at the risk in the terms of, well, here's the disasters we're familiar with. You know what?

The chances of getting that smoking whole event are so low. I'm going to go ahead and accept that risk. But the chances today of being ransomware. I mean, it's basically going to happen. Right. It's going to happen every minute. So if you if you've accepted the risk because the statistically, you know, the natural disasters won't get you and you'll probably be OK. The risk calculations change for ransomware. So it gets a lot harder to actually accept that risk for many organizations, even though that's what they're used to doing.

So they've never planned for anything. So what are some of the typical things that people don't account for? Because I see these stories in the news all the time. Companies that should have great backup strategies. Is it just a is it just a issue of not having a great backup strategy or not specifically thinking about how to recover from ransomware? It's a little both. One thing I like to say is that ransomware is a disaster. So this really ties back to a lot of the traditional DR planning, which is easy to blow off, because like you said, well, I'll have a smoking hole event.

I'll be fine. I know I can recover. It'll take me a little bit of time. But whatever. Right. The chances of that happening are so low. Like I said, the paradigm shifts with ransomware. We're back in a place where, OK, we know this event is going to happen. But guess what? We don't even have an inventory of our assets. So we're not sure we're backing everything up. We haven't done a business impact analysis in years.

So our RPOs and RTOs that we're trying to meet are probably not even sure if we can meet them. Are they even accurate? So all this poor kind of business continuity disaster recovery hygiene of recent years is kind of coming to light now. So really, the time is now that you need to go get everything into order so you can recover later. So let's peel back on that a little bit. So if I if I want to start the process of reevaluating my DR plan or test, where should I simply start?

Like where where's a good starting place? Because it looks like it's overwhelming. It is overwhelming. The first thing is, if you have you already have a disaster recovery plan, go ahead and test it. Right. At least find out the flaws and what you have today. So either it works or it doesn't work, or maybe you can restore a couple servers, but you can't restore a scale. If you have something, go test it and make sure it's working. If it's not, now you have time to fix it, but then go back a step, then start with like the asset inventory.

Right. The basics. Do I have an inventory of all the assets I need to protect? Because you might not. Right. If that's not something that you've automated, you might not have an inventory of everything, which means you're not protecting everything. So the next thing is to get the big picture of everything and at least back it up like once a week, once a day, whatever. Get it protected, even if it's not quite the right RPO yet. Get that stuff protected.

And then it becomes a bigger task where you really need to work with the business owners and say, OK, let's go app by app and figure out what does this do for our business? What is the consequences of downtime and how do we protect this application or this data accordingly? Yeah. And I would like to add, have a process to maintain this. This is where I failed in my own data center. We had a inventory of all the critical assets.

So I was able to recover Active Directory, DNS, the just critical infrastructure assets. Then a couple of weeks later, we have contractors that come in and do things in the data center. They they're doing. We have researchers who are running projects. And one of the researchers came and said, hey, Keith, when are you going to restore my my project? And I'm thinking, what project are you talking about? And I missed a process. I missed the process of of.

Bet inventorying when we have various research projects going on. And again, the process of reassessing my risk profile during those projects. So, yes, the CTO advisor data center as a big fifty thousand foot view can. The business will recover from being down two to three weeks from a ransomware attack. That's not a big deal to the overall business. But if I have a project that's going on and I have a tight deadline and that tight deadline is disrupted because of a ransomware attack.

And now I have to not just recover the data center, but I lose all of the research and work that was done. My business is a bigger problem from that. Yeah. Well, there has to be some process, some business process that reevaluates consistently, reevaluates risk and a technical process for how do you inventory that? Exactly. And that's the tricky part, because when we talk about this kind of stuff, it's not one person or one team in an organization, right? It's it's everybody.

It's the security team. And a lot of times BCDR does fall under security, right? It's the infrastructure teams. It's the apps teams. It's the backup teams. It's it's everybody kind of needs to be on the same page and work together. And just from an organizational standpoint, that can be really hard to do. So when you engage customers, when you're talking to customers about ransomware, is this mainly activity that's driven by, you know, to your point, is this the activity that's driven by InfoSec?

Who are you talking to inside of the organization as you're helping them get the proper hygiene? You know, I've noticed a huge shift over the last year where now I am talking more to the InfoSec people than I ever were before. And I can't tell you how happy I am to see that shift because now they're finally understanding. I've been saying for years, this is a security thing. This is a security thing. This is a security thing. And it seems like people are finally catching on to that.

And most of the engagements I've done lately really are a good, holistic group with representation from all across the organization. So I think people have been scared enough that they're starting to get it and they're starting to put their houses in order. So can I give you one more really scary thing about ransomware? I don't want to be scared anymore, but let's go. I'm sorry. I'm sorry. Now, this is something I have to talk about because I'm very, very heavy in the VMware ecosystem.

I might have went and read some of the playbooks from ransomware organizations that were leaked. Right. Because I usually do that when that kind of stuff comes out. There are detailed instructions on how to compromise vSphere in there from vCenter to the host themselves. They are walking through these hackers who have gotten into your environment. Here is how you destroy VMware. Right. Because it's high impact. Right. If I can get onto one host, I can encrypt that whole cluster because every data store is connected to that host.

I can down them really, really fast. So if we're talking about high impact for a ransomware actor, they're heading right to VMware. They're heading right there. Yeah, that is scary. That kind of gets to the title of this episode. Is Linux a target for ransom? We usually like desktops and endpoints, especially Windows. Right. I like to joke about RDP or ransomware deployment protocol. Like a lot of people have this mentality that, no, it's totally a Windows thing, but it's not.

It's Linux. And eventually the Linux evolves into ESXI. Yeah, we slap people's hands when they have RDP enabled publicly. But the same people who slap the hands of the folks that have RDP enabled publicly, we don't think twice about SSH. Somehow SSH is somehow better than RDP. It's the same thing. Right. And one of the biggest things is I was just like looking at stuff people were searching for. I saw people were looking for how to enable ESX on it, how to enable SSH on ESXI.

I'm like, that's great. But I hope everybody knows that you should immediately disabled it when you're done with it. Because if someone gets in there and SSH is open on the host. Right. If I can get credentials that I can SSH in, you're done. You're dead. You're done. You're wrecked. So as we're thinking about this and we think about all the machinations of ransomware and what can go wrong, let's have a let's let's end on a happy note.

Yes, absolutely. Where have you seen ransomware recovery kind of go right? And it is one of those. Oh, here's the ransom. You know, you must pay the ransom. And it's kind of like, I mean, I just inconvenienced me. I've got a funny story. And this will be a little anecdotal story I guess to end with. Last year, I was doing a lot of work in the public cloud and I got ransomware. Myself, me, myself, I got ransomware because I had RDP open.

I didn't care. I was just trying to get this lab working. And my backup server got ransomware, which was connected to my backup repository, which was ransomware. Right. So I'm sitting here and I actually started laughing when I saw this because I had another copy of my backup data someplace else. I used a different server. I connected to my backup repository. I rescanned it and I was restoring in like six minutes. And that was just like me setting up my silly little lab.

I still had the force. I'd be like, I need an offsite copy of my data in case something goes wrong here. And I was up and running in no time at all. So the good news is with a little up from planning, as long as you're willing to be honest with yourself and what your environment is like today, there are simple things you can do to enable your ransomware recovery later. Right. For me, I do believe in good security practices, of course, but you still need that plan at the end of the day.

If they are in, you are almost better served spending your time making sure that you can recover when it happens than just solely focusing on not letting the hackers in. And then, well, if they get in, I don't know what happens next. Right. Let's start with the worst case scenario. Pretend they're in. How do you go in and make sure so you can recover later, like do that stuff now. And then from there, once you know you can recover it now, let's go back to kind of the security hygiene and try to prevent them from getting in.

Yeah, we talked to some principal architects over at HPE, some principal engineers. They're, you know, they're the fellow level folks and they were talking to us about some of the advanced capabilities in ransomware. Ransomware attackers, to your point, are specifically going after backup infrastructure. They are smart. They will put time lapsed or time bombs into environments. They could be in there for months and you have no idea. They know when you're back up. They know your retention policy, your backup policy, so you don't know when to recover to.

Even if you did recover, it is going to be encrypted anyway. Another nasty one that they that that I've seen that I heard them talk about has been that they'll just change a one K block. So that you won't you your your intrusion protection system won't get triggered. But if they only change a one K block, that's just a regular right. And you won't notice any inactivity at the storage level. And it's the same impact. Randomly encrypting one one K blocks at a time still corrupts an entire database.

Exactly. So this is nasty. If if you if you don't think you need to pay attention to ransomware, you're wrong. If you're wondering where you should start, go to be missed that net. That's a good start. Melissa, thanks for joining the podcast again for folks that want to follow you. You're you're doing great work over at the Web site. We miss that net. And if you're on Twitter, Melissa is very much part of the VMware community. 33.

If you want to learn more about the CTO advisor, you can follow me on the Web. The CTO advisor dot com is the Web site at CTO advisors. The Twitter handle DMS are open if you have questions about ransomware that Melissa was unable to answer for you. I'm not going to be able to answer them for you, but I agree. Let me know and I'll let you know and she'll find you the answers. Thank you so much for having me, Keith.

Thank you for being on. Talk to you next. CTO advisor podcast.