Coronavirus Disaster Recovery - CTO Advisor Episode 116

Machine-generated from the episode audio and not hand-corrected, so names and technical terms may be imperfect. The audio is authoritative.

Hey, you're listening to episode 116 of the CTO Advisor Podcast. We have a returning guest, Joey DeAntoni, Principal Consultant at Denny Cherry and Associates. Joey, welcome back to the podcast, man. Hi, Keith. How's it going? Well, you know what? I have to, with all things considered, I have to say it's going pretty well. S. S. phenomenon. We have this coronavirus or better now labeled COVID-19 issue going around. How are things in your world and how is it affecting you, if at all?

Well, I had my first thing get canceled this week, which is next week I'm a Microsoft data platform MVP, and we were supposed to have our annual MVP summit, which is they fly all the MVPs out to Redmond to meet with their specific product groups. And that was canceled this week or excuse me, not canceled, but moved to a virtual conference. So it'll be online and like numerous other conferences have been in the last week or two because of the concerns about COVID-19.

Yeah, I know your role is similar to mine's where it's very conference heavy. I was headed to Tech Field Day 21 next week and that was moved from in-person to virtual. You know, we're scrambling and doing an in test in kind of in the moment test, I want to call it like our kind of last minute getting the crews together and testing Zoom, etc. Lighting and doing these things we're not accustomed to for doing tech field days. So that's been interesting.

And then everything else that I have up to the beginning of May has been canceled, KubeCon 2019 or 2020 in Europe. So much so that I decided to even host my own conference. I'm doing a virtual conference, April 21st. I saw I saw that. That sounds kind of interesting. Well, interesting is not the word for it. I've supported these for the VMware user group for the past couple of years. And a virtual conference, I mean, a regular conference is a lot of work and a virtual conference isn't much less work.

You don't have the logistics of the local event and all of that, but scheduling speakers, etc, etc. It's all still a lot of work and lining up sponsors. I'm I'm kind of regretting it, I guess. But, you know, my business is to is a good portion of it is to do conferences. So, you know, we all have to make adjustments. Yeah, for sure. I read an article this week for Redmond Mag. I have a monthly column there and kind of talking about some of the effects of this on, you know, regular businesses.

Not necessarily like what we do or technology, even technology organizations, because that was something else that happened this week. Microsoft basically sent everybody home on I believe it was Monday night or Tuesday. And, you know, said at least basically the end of March until we reassess everybody work remotely. So that is kind of crazy when you think about it. Yeah. How many employees does Microsoft have? I believe in it. So it was specifically for Redmond campus and San Francisco campus.

And I want to say Redmond campus is like forty five thousand. But I it's it's close enough that, you know, they're that's a reasonable number. So forty fifty thousand people. So that whole exercise getting get me to thinking that we're basically. R. exercises, what I'm saying, oh, what it was a pandemic. And that's kind of been a throwaway term, I think, in a lot of businesses that consider pandemics as part of their business continuality plan. It was really funny.

I was on a slack. It actually was a tech field day slack channel. And I said, man, this covid is really going to impact my revenue from a conference perspective. R. plan. R. for, you know, most of the conferences being canceled for at least the first half of the year. Yeah, for sure. I know. R. planning for a major pharmaceutical company, actually, that no longer exists. I'll name them why they're in by Pfizer now.

And we have all the companies where I've ever worked. R. planning just anywhere I had worked. They put a lot of time and thought and spent a lot of money on it and did testing and things like that. And as part of our business continuity plan, we did have something about pandemic. But, you know, it was like a throwaway line that, like, everybody will work from home. But I can tell you, I can guarantee you our VPN appliances were not sized for everybody working from home.

We didn't think about, you know, special types of employees and they have special requirements to work from home. T. people would be on call and maybe, you know, five percent of the company to be working from home somewhat regularly. So I threw that out there on Twitter, kind of a short three minute video. If. Your CEO came to you and said, hey, you know, Joey, tomorrow, the 80, 70, 80, 90 percent of our workforce, all of our knowledge workers need to work remotely beginning tomorrow.

Kind of like what Microsoft did, like, you know, at the Redmond campus, you know, Monday evening, everyone go home and you're just working from home the rest of the week. And then they extended it to the rest of the month. Like, let's walk through that mental exercise. What would have to happen? So for Microsoft, it's easy, but they're a software company and they're built around having employees globally. But if I'm an enterprise and I had my hope is that I have some presence in a public cloud.

Right. And I can I can spin up some kind of scale at virtual desktop solution or I have a large scale virtual desktop solution in my data center. But the chances of you having a virtual desktop solution that can support all of your employees, that's a big challenge. That's just the first thing. So you have to have just the sheer computing capacity. And then not only that, you have to have profiles where you have, you know, the right software installed on those remote desktops or be able to hook it into your configuration management software to push the right things for the right types of employees.

And it's something if you haven't thought about, it's going to be problematic. Another solution, you know, if you're a smaller company and when I say small, I mean probably less than 100 employees. You go to the store and you make sure everybody has a laptop that if you have more than that and you wanted to do that, that's challenging. Right. Because where did this virus start? Asia, China. Where are most computers made? China and Taiwan. So there's a pretty major supply.

We had to order a new laptop for one of our consultants and unrelated to this, hers just died. And it took about an extra three weeks to ship just because of the delay is coming out of Asia right now. R. There's just the nuances that you just don't think of, like if the supply chain is disrupted, how can you get capability? R. projects in your past. So, you know, you have companies that have everything from hot spare sites ready to go hot cut over all the way to people saying, you know what?

We'll ship the tapes to Cincinnati and we'll buy equipment, you know, that day that we will be up in a week. And for this pandemic, it feels like most of us are in the situation where we're that we're that company is said, oh, well, we'll ship the tapes to Cincinnati and we'll. And we'll buy equipment like but now we're practically looking at it. One of the scenarios you gave me for, you know, specialized office workers is what if your business is a call center?

And you have employees that have they can have VoIP headsets, you know, Cisco VoIP or via VoIP physical headsets. But how do you move that from a call center to a virtual home office based setup if you've never done it before? Yeah. Yeah. I mean, the other thing, too, these are pretty bandwidth intensive things. Right. So you if that happens, your employees would have to have enough bandwidth in order to just share Internet bandwidth to get out powerful enough laptops to run that.

You probably need some kind of Bluetooth thing or USB port. And who knows how that's going to work in terms of working through a virtual device? Like if you're using, you know, VMware's virtual desktop or Citrix, unless you have a fully certified solution, you only know what laptop you're potentially going to be running on. And we haven't even started talking about the notion of throwing a whole bunch of remote desktops out on the Internet where they can be accessible. So you really need to stand up two factor authentication for all your employees in front of that and train them on it.

Yeah, that is getting even worse. So my daughter worked from home as a call center agent and she had a company issued laptop. She had a company issued Internet gateway that established a small office VPN. She had access control on her laptop so that she could only only her laptop was authorized to be connected to the Internet gateway. It was a really well thought out solution. They had already had a AT&T DSL provision by the company. So they essentially managed bandwidth and connectivity whenever she had connectivity issues, a AT&T tech business tech would come out and troubleshoot.

So this was, you know, well thought out months, if not a year in advance so that you can give remote workers these capabilities. You know, I simply said one of the things that spurred this conversation is that one of the challenges that I can definitely base a recommendation on is that you can't change the way you operate in a DR from when you're operating in production. At least as much as possible. A lot of people, you know, they kind of respond. Oh, I just simply go out to AWS and get a bunch of desktop.

One of their AWS has a desktop as a service offering, which is pretty decent service. Yeah, but it's so different than if you're using Citrix frame or even VMware Horizon. They are completely different solutions to operate on the back end, especially with non-technical users. Right. So you're dealing with non-technical users who don't have a help desk that they can call necessarily or help this person. They can get some assistance from. You know, this is causing an IT supply chain issue now for people like us.

Are Amazon, Microsoft and Google going to have a supply chain problem in about six months? Because you figure you figure they're buying enough capacity ahead of time that they're staying on top of their capacity. But if there's a slowdown coming out of Asia in six months, is there going to be adequate cloud capacity? So, like, if you're deploying a new project there, might you want to start getting your resources now and just eating the money just so, you know, you know, they're there if it's an important project?

Well, this sounds a lot like how we did it pre cloud. But here's a practical, you know, example of that. Again, I mentioned that I support supported VMA for their virtual events. So I already knew what company to go to for to host my virtual event because you need a virtual event platform. com. They didn't sponsor us or I wish they would give me a break off the price. But typically you could go there and get an environment set up and up and running and and lead time to actually host your event would be about two weeks.

I talked to them yesterday and they said the lead time was four weeks. Now, it doubled because obviously everyone canceling their in-person events, they need they don't already have a virtual platform. Have they have to build one or they have to build or purchase will rent one. And they're doing it from the bears. And they said they can't hire people fast enough enough to keep up with the demand. So how much worse will this capacity issue get for cloud providers?

Because in normal times, they may purchase for six months out. But if they're getting additional demand from the zooms of the world, the B bears and all these SAS services holding hosting back ends, they are now creating additional demand. Will that crunch come sooner? Maybe they can pull some of that capacity from the airlines that are shutting down stuff. The problem is that most people don't need a 400 capacity. That's that's that's the problem. No. So let's talk about let's leave the listeners on some positives on some things that they should actively be doing.

If you were looking at the typical enterprise and besides the work from home challenge that we talked about, what are some of the things if they haven't started considering now that they should consider? And what are the things that they really need to put on their whiteboard to look at in the next few weeks? I feel like I've been yelling this for three years because of ransomware issues. I feel like multi-factor authentication for everyone and getting that rolled out. If you can get that rolled out while your employees are in the office, it will be a lot less painful for everyone because you can assist them with it.

And you and I were talking about this on Twitter the other day, and I think we disagreed a little. The configuration is easier than it's ever been, especially if you're using something like Google or Microsoft Authenticator. You usually just take a picture of barcode after going to a site. It's still user training and you still need apps that can support it. But it's definitely something I would that would be one of the first things I'd get because it's something for any other solution you like that you need.

You need a place for any sort of virtual desktop solution. If I didn't have a public cloud presence for pick-your-flavor virtual desktop that you use, I would start looking at getting one just so you can scale your capacity in case you can't scale it in your own on-prem data center. And then the other thing is you just got to communicate with your employees. And even if you're not in a situation, and this is something where, like, as an IT manager, you need to be working hand-in-hand with the senior management of your company because your employees are seeing all this in the news, and they don't want to be in the dark.

And people are paranoid about, you know, if I get sick, I might have to take two weeks off and not get paid for it. Or even if I don't get sick, even if I just get exposed, you know, do I have to take two weeks off and not get paid for it? So communicating with your employees as to where you are in the plan and what you're thinking about is definitely something I would recommend. Yeah, I love those suggestions. I'd add one other one to that is get a cross-connect to a cloud provider because you're simply just not going to have enough bandwidth to service all of your virtual desktops, even if you could scale the physical capacity inside of your data center.

Having that number of random Internet connections from all over either the region or the world coming into your data center for VDI, probably you're not going to be able to get enough bandwidth, in my opinion. If you have a cross-connect into the data center, let the public clouds take the brunt of servicing those desktops, et cetera, and then just have the connectivity back to your data center to access the Fed apps, et cetera, et cetera. You're going to experience bottlenecks and challenges no matter what, because this is a skill of challenge that we've not seen before.

And I don't think we really disagree much, if any, on the multifactor authentication. This is a great time to roll out multifactor authentication, get everybody deployed. I think what you'll end up problems with is one user training is not just you. It's not just user training is user acceptance because I resisted putting a work related app on my phone. So even if it's Google authenticated, which isn't necessarily work related, my job is asking me to bear the burden of having a token.

And I just I don't like that idea. And I'm a technologist. So. You know, common office workers who are not used to working from home, they need to be educated on this stuff and they need to have buy in your. I love your point again about communications, training users on how to work from home, not just from a technology perspective, but from a practical perspective. What does that even look like? Can I walk my dog for my 15 minute coffee break?

You know, just coach. How do I keep the culture of my company intact? So there's just a a a smorgasbord of problems related to this pandemic. So let's leave it with this. Joy, how are you hunkering down on this? You know, are you working from home permanently? Like, are you taking meetings? What is your. Well, I normally work from home, but I am working from home. K. and actually I'm traveling to Chicago in two weeks for a SQL Saturday.

K. to teach an Azure class, but it's pretty small. So I could see that continuing on. It's I think 10, 15 people. K. S. does at the moment. So that makes me feel a little bit better. Well, Joy, I'm wishing you the best of luck, the best of health for you and your family. If people want to reach out to you, ask you questions about some of the information you've done or reach out to the company.

How can they get a hold of you? They can either hit me up on Twitter. I'm Jay Dan Tom there or they can email me at Joey at DCAC dot com. That's it for this episode of the CTO Advisor. If you want to ask me questions, you can DM me on Twitter. I'm at CTO Advisor on Twitter. You can find the website, the CTO Advisor dot com. By the time you hear this, the registration for the CTO Advisor virtual conference will not be up.

But the virtual conference is happening on April 21st. We are looking to just have great, great content. The link to if you think you might have a topic you want to talk about, you can. The link for the call to papers will be up until the middle of the week of the publishing of this podcast. I'd love to hear from you. Until then, rate us on iTunes. Share us with your friends. Talk to you next CTO Advisor podcast.