On July 19, 2024, at approximately 4:09 AM UTC, CrowdStrike released an update to its Falcon Sensor, one of the most widely deployed endpoint security tools in the world. By the end of the day, 8.5 million Windows systems had become completely unusable, rendering them unbootable with a blue screen of death. That’s not a hyperbole. Hospitals postponed surgeries. Airlines canceled flights. Traders couldn’t access their trading floors. Banks closed branches. Restaurants that rely on point-of-sale systems went down. Police departments that depend on databases lost access to critical records. The outage wasn’t caused by ransomware or a sophisticated attack; it was caused by a single file deployed in a security update that had a bug.

This is the kind of incident that gets embedded in everyone’s memory because it breaks the assumption that you can trust your security vendor. CrowdStrike is a well-respected company. They have serious security practices. They have thousands of enterprise customers. And yet a single update, without proper testing or staged rollout, took down the global economy in ways that are still being measured weeks later. If you’re working in IT and you didn’t immediately think “could this happen to us?”, you probably should have.

The Technical Failure: What Actually Went Wrong

The root cause of the CrowdStrike outage was traced to a content update delivered through Falcon Sensor’s normal update mechanism. The update file, which was supposed to contain sensor content rules, had a malformed data structure that caused the Falcon Sensor driver (which runs at kernel level on Windows) to crash. Because the driver runs at ring 0, a crash in the driver triggers a blue screen of death. Because this deployment was automatic and global, the blue screens started happening on millions of machines almost simultaneously.

What’s telling is how this happened. CrowdStrike has automated systems for pushing updates to customers globally. The update went through testing, but apparently the testing didn’t catch this particular failure mode. The content update didn’t require a kernel driver restart to apply; it was supposed to be a hotfix that didn’t need reboots, which likely made the testing process feel safer than it should have. And critically, there was no automated rollback mechanism. Once the bad file was deployed, it couldn’t be quickly pulled back.

From a pure technical perspective, this is a relatively simple bug. The kind of thing a developer catches during code review in a company that has a strong development culture. But it got through to production on 8.5 million machines without anyone catching it first. That’s a process failure, not a technical one.

Why This Exposes a Vulnerability in How Organizations Handle Change

If you’re managing infrastructure, the immediate lesson from the CrowdStrike outage is about change management, and specifically about how much trust we’ve placed in automated updates from vendors without enough scrutiny.

Most organizations have policies about who can deploy changes to production systems. You probably have change boards, approval processes, testing requirements, and staged rollout strategies. But when it comes to updates from vendors like CrowdStrike, or Microsoft, or other established software providers, a lot of organizations have disabled those controls. The reasoning is that these are security vendors, or operating system updates, or things that are categorized as “vendor responsibility,” so the standard change management process doesn’t apply. You enable automatic updates and trust that the vendor’s testing process is sufficient.

The CrowdStrike incident is a hard demonstration that this is a bet you shouldn’t make, no matter how reputable the vendor is. A proper change management process would have required that even automated updates go through a testing window before full deployment, that there’s a canary or staged rollout, that there’s an automated rollback capability, and that there’s monitoring to catch failures quickly. CrowdStrike was doing some of this, but not all of it, and the ones they were doing weren’t sufficient.

The second the CrowdStrike update hit, the organizations that had the most resilience were the ones with the most caution about updates. Some companies had disabled automatic updates and were manually testing new versions before deployment. Some had made endpoints update in waves so that if something went wrong, not all machines went down simultaneously. Some had infrastructure that was diverse enough that not everyone was running CrowdStrike, or was running it alongside other endpoint detection and response (EDR) tools that worked independently. Those organizations were disrupted, but they weren’t catastrophically disrupted.

The organizations that got hit hardest were the ones that had optimized entirely for “vendor is trustworthy, updates are automatic, we don’t need to think about this.” That optimization is actually a vulnerability, and it’s the kind of thing that a good IT team thinks carefully about.

The Vendor Trust Question Gets More Complicated

This incident also creates a conversation you probably need to have with the vendors you trust most. CrowdStrike is a security company. Security is literally their job. They failed in a way that affected millions of customers. The question isn’t whether to continue working with CrowdStrike (they’re an important vendor with otherwise solid practices), but rather what you require from vendors going forward and how you monitor whether they’re actually meeting those requirements.

If you’re responsible for vendor selection or management, you should be asking some hard questions. What’s the vendor’s process for testing updates before they reach production? Do they have a canary deployment process? Do they have automated rollback? Do they have rapid deployment that can address critical issues within minutes if something goes wrong? What’s their communication protocol when an issue is detected? Do they have service level agreements around detection time and remediation time?

For most vendors, the answer to some of these questions will be “not as rigorous as you’d hope.” That doesn’t mean you can’t work with them, but it means you need to build your own safeguards around their updates. You don’t trust them blindly; you trust them with guardrails in place.

This is also a conversation that probably needs to happen with your executive team and your change management board. The standard approach of “vendor updates are automatic and we don’t need change control” is actually a security risk disguised as efficiency. Proper change management for critical vendor updates takes more work and more planning, but the cost of not doing it is the CrowdStrike scenario: all your machines down simultaneously because a vendor update broke something.

What Incident Response and Recovery Look Like at Scale

The CrowdStrike outage is also a clinic in what happens when incident response hits a system at the absolute largest scale possible. CrowdStrike’s response team was excellent, but they were fighting against the reality that their ability to fix the problem was constrained by the fact that the machines that needed fixing couldn’t boot. They couldn’t remotely deploy a fix because the machines were stuck on a blue screen. Recovery had to happen manually, on each machine, either by having someone physically access the machine or by remotely booting into safe mode and manually removing the problematic file.

For machines in data centers, that meant IT teams had to physically access servers or use remote management interfaces (lights-out management) to recover. For end-user machines, it meant either remote troubleshooting sessions or, in many cases, users having to bring their machines to IT for recovery. Multiply that across millions of machines, and you’re looking at a recovery effort that takes days.

This is why resilience is non-negotiable. Organizations that had spare capacity, redundant systems, or machines running different operating systems suffered far less disruption. Airlines that still had the ability to manually handle some operations survived. Hospitals that had paper-based backup processes for critical systems could keep running. Financial institutions that maintained manual trading capabilities didn’t have to turn customers away. The organizations that had designed for “something critical could go down and we’d still function” weathered the storm. The ones that had optimized everything for zero spare capacity and total dependency on automated systems got hammered.

The Lesson for Your Incident Response Program

If you’re responsible for incident response or business continuity, the CrowdStrike incident should prompt some hard questions about your own organization’s resilience.

Do you have the ability to detect when something is wrong with a critical vendor’s update? This probably means monitoring not just the vendor’s status page but also your own systems for anomalous behavior. Alerts for unusual restart patterns, boot failures, or system crashes should probably be tuned to catch something like this.

Do you have the ability to quickly roll back a vendor update if it breaks something? This probably means maintaining golden images of your systems, having versions control on critical vendor software, and having a tested recovery procedure. If an update breaks your systems, can you get back to the previous version without manual intervention on millions of machines?

Do you have the ability to function if that vendor’s service or update process becomes unavailable? This doesn’t mean you need to replace CrowdStrike (or whoever your critical vendors are), but it means you need to think about what “degraded mode” looks like for your organization. What can you turn off safely if you have to? What can you operate manually for a while if you have to?

These aren’t easy questions, and the answers typically involve building redundancy and resilience that seems wasteful until the moment you need it. But the CrowdStrike incident is a pretty clear demonstration that the moment you need it, you’ll be very glad you have it.

Career Implications: This Is Where People Get Promoted

The IT professionals who got promoted after the CrowdStrike incident were typically the ones who either prevented the problem from affecting their organization or who responded effectively when it did. The prevention story is usually someone in change management or infrastructure who had the authority and the relationship with leadership to require vendor updates go through a testing process. The response story is usually someone who had the technical knowledge and calmness under pressure to guide teams through recovery.

If you’re thinking about how to make yourself valuable and promotable, this incident illustrates where that value actually lives. It’s not in keeping systems running smoothly under normal conditions; anyone can do that. It’s in thinking clearly about what could go wrong, building safeguards, and then being able to respond effectively when those safeguards face a real test.

Specific skills that matter: change management expertise, understanding how to think about vendor risk, the ability to design and test recovery procedures, incident command training, and the kind of calm, methodical thinking that lets you solve problems when stakes are high. These are skills you build by looking at incidents like CrowdStrike and asking “what would I have done?” and then actually building the processes and tools you’d need.

What You Should Do Monday Morning

Start with your change management process for vendor updates. Specifically, for your most critical vendors (endpoint protection, cloud infrastructure, identity management, etc.), ensure that even automatic updates go through a testing phase before they hit all your production systems. A testing cohort that’s representative of your environment but small enough that you can recover from quickly is worth the extra complexity.

Then, talk to your vendor contacts at companies like CrowdStrike. Ask them about their testing process, their rollback capability, and their communication protocol for critical issues. Their answers will be informative about how much caution you actually need.

Finally, look at your incident response program and ask whether it could handle a situation where a critical vendor update broke millions of your systems simultaneously. What’s your manual recovery process? How do you prioritize? How do you communicate with users and executives? What’s your absolute minimum to keep the business running? These questions don’t have easy answers, but asking them and thinking through responses will make your incident response program materially better.

The CrowdStrike incident was a shock because we’ve outsourced a lot of critical infrastructure to vendors and we’ve stopped thinking carefully about what happens when those vendors make mistakes. The response isn’t to trust vendors less; it’s to be smarter about how you integrate them into your systems, how you test their changes, and how you prepare to recover when something inevitably breaks. That’s where your value as an IT professional actually lies.

For more on how critical incidents reshape organizations, see The SolarWinds Breach Changed Everything and Zero-Day Crisis Management: The Soft Skills That Companies Pay For.