What can we learn from the CrowdStrike Windows outage?
Transcript
they say all publicity is good publicity but I'm not sure that crowdstrike wanted quite as much publicity as they got last week pushing out a simple profile update a data file update to apparently 8 and a half million Windows machines caus those machines to blue screen to catastrophically fail this is probably not the best thing to have happen on a Friday it affected Airlines it affected supermarkets here it affected even our online fpos here in New Zealand as well definitely a high impact uh situation and
one of the big challenges with it is due to the nature of that failure the blue screen it's not just that the software failed the entire operating system failed every single affected computer had to be visited in some way manner or form this you couldn't automatically deploy out of replacement to this so this was a big deal and we'll probably still hear some ripples for a while of those 8 and 1 half million Windows machines that failed a lot of credit to crowdstrike themselves and particularly
to the CEO George Curts he showed up on USA Today after what must have been a pretty horrific night with himself and his team working out what the heck had gone wrong and how they were going to handle it uh he owned up and said it's definitely it's an issue of of ours it's not a Cyber attack it's not a compromise it was an an error uh really appreciate the honesty and the directness of turning up there not a situation any CEO wants to be in
and I think George did a really good job what it highlighted was that there's some tension between the speed of response to new threats and that's why this update was released there was a new uh attack that was using a particular functionality called named pipes that only exist on Windows machines and that's why this fault only occurred and only only the outage only affected Windows machines cuz that's the only place name pipes exist crowd strike had released an update to specifically protect against this compromise that
was underere exploit and that uh release had to be pushed out very fast in order to protect customers and there's that tension going on then about speed of release versus safety of release clearly this code was not safe to release into production because caus such a widespread outage and that's a tension we see in a devox practice of that tension to to get changes out rapidly against the need to fully test changes before they're released out it's important to to think about building those cicd pipelines
and your devop practices around using sufficient testing and testing that reflects what's going on in production if during the testing of this change uh crowd strike had found that it caused blue screens on Windows machines they would have rolled this back they would not have released it out into production now that tells me that what they're using in their testing doesn't match up to what's out in customer's actual production and that's a a real problem for testing because you've got to test what's out in production
because production is the ultimate test environment that's where your code has to actually work uh one of the mitigations that we sometimes use in an Enterprise use case is a segmented deployment where you only deploy to a few machines and you wait a while and you deploy to a few more because from what crowd strike have reported the faulty code was only available on their download server for a period of around an hour and yet it got to 8 and a half million machines uh there's
some challenge in there around that segmented deployment I suspect what's gone on is that each customer has downloaded one copy of this update from the crowdstrike servers and then proceeded to deploy it out to every machine that that is managed within that customer so that sort of stage deployment and I suspect crowdstrike don't have a way of recalling an update once it's been sent out so that update comes off their download server goes to the customer staging server uh what they don't appear to have or
what I suspect they don't have is a mechanism for that staging server to be notified not to keep deploying that update and that's kind of would have helped here I suspect however one important perspective is that although 85 million machines were sh shut down what if 8 and a half million machines have been compromised by the attack that crowd strike were protecting against maybe not 8 and a half million would have been compromised maybe a few could have been compromised in that time but this is
the the tension that we see in security products that arms race between attackers and our own defense systems and so sometimes things go wrong and because crowd strike is such a dominant player in this area uh when it goes wrong it goes wrong very publicly in your own cicd practices make sure that you are doing thorough testing of releases before they go under production most of us don't have that speed of uh working against threat actors that we're really thinking about working against other businesses and
being competitive and so that speed is not quite so critical the safety of release fully testing packages segmenting your deployments out really good practices and develops and it's important we don't just focus on how many deploys per day can we get out when we're implementing devops we really want to be thinking about the safety and quality of those uh deployments our objective is to rapidly get code safely into production and without proper testing it's not safe I'm Alis cook CTO advisor here at the futurum group
stay tuned for more CTO dose
