Lessons Learned from the CrowdStrike Mistake

The CrowdStrike mistake was a costly one, with the company losing $15 million in a single day due to a massive stock sale.

CrowdStrike's stock price plummeted after the company filed a Form 4 with the SEC, revealing that several executives had sold millions of dollars' worth of shares.

The SEC filing showed that executives, including CEO George Kurtz, sold over 3.5 million shares in a single day, accounting for nearly 20% of the company's outstanding shares.

This sudden and large-scale sale of shares sent a negative signal to investors, leading to a sharp decline in CrowdStrike's stock price.

A different take: Crowdstrike Price Prediction

The CrowdStrike mistake was a perfect storm of errors, caused by a mismatch between input parameters and predefined rules in the update. This mismatch led to the Falcon sensor's rules engine not understanding the new threat detection configurations.

The issue arose on July 19, 2024, when new threat detection configurations were validated and sent to sensors running on Microsoft Windows devices. The configurations were not understood by the Falcon sensor's rules engine, causing the sensors to malfunction.

For your interest: Crowdstrike New

The problem was not just a simple mistake, but a result of our reliance on a homogenous system that can't be perfect all the time. These digital systems are designed to safeguard our sites, but they are going to fail at some point, whether from deliberate attacks or a simple mistake.

The incident highlights the issue of digital monoculture, where decades of anti-competitive practices have created a system where one mistake can have a big impact. This monoculture leaves us incredibly vulnerable to the reason the patch was developed in the first place.

Broaden your view: Fix Written Mistake

CrowdStrike's efforts to prevent similar incidents in the future are noteworthy. They've introduced new validation checks to ensure that the number of inputs expected by the sensor matches the number of threat detection configurations provided.

These validation checks are part of a broader effort to improve the overall quality of their system. CrowdStrike has also enhanced their testing procedures to cover a wider range of scenarios.

To give customers more control, CrowdStrike now allows them to deploy configuration updates at their own pace. This is a significant improvement, as it allows customers to implement updates on their own schedule.

Here are the specific measures CrowdStrike has taken to prevent similar incidents:

CrowdStrike's efforts to resolve the issue were also noteworthy. They introduced automated techniques to accelerate remediation and deployed staff to assist customers in recovering their systems.

The incident that silenced so many systems is a great example of cascading failures that can occur in our relatively homogenous systems.

We rely on these digital systems to safeguard our sites, but they're going to fail at some point, whether from deliberate attacks or a simple mistake.

The problem is that we're stuck in a digital monoculture, where one system is responsible for so much of what we rely on, from airlines to hospitals to schools.

One mistake that creates a big failure is an inevitability, but for it to have this sort of impact is a policy failure.

The patch was developed to fix a problem, but the very system that was patched left us incredibly vulnerable to the reason it was developed in the first place.

CrowdStrike's efforts to reboot affected systems involved introducing automated techniques to accelerate remediation and deploying staff to assist customers in recovering their systems.

Physical access was required to an affected machine to reboot it, which made the process more challenging.

Fortunately, virtually all of CrowdStrike's customers' systems were back up and running by July 29.

Expand your knowledge: Is Apple Affected by Crowdstrike

CrowdStrike has taken significant steps to prevent similar incidents from happening again. These improvements include validation checks to ensure the number of inputs expected by the sensor matches the number of threat detection configurations provided.

The company has also enhanced its testing procedures to cover a broader array of scenarios. This means they can identify potential issues before they become major problems.

CrowdStrike's customers now have more control over the deployment of configuration updates to their systems. This gives them a sense of security and autonomy.

A phased approach is now used for rollouts of threat-detection updates, allowing customers to implement updates at their own pace. This reduces the risk of sudden changes causing issues.

Additional runtime checks have been added to the system to ensure data matches the system's expectations before processing occurs. This prevents errors and potential security breaches.

Two independent third-party software security vendors have been hired to review CrowdStrike's Falcon sensor code and end-to-end quality control and release process. This adds an extra layer of security and accountability.

Here are the measures CrowdStrike has taken to prevent similar incidents: