CrowdStrike, a leading cybersecurity firm, experienced a global outage that left many of its customers without access to its services. The outage was reported on multiple continents.
The issue was widespread, affecting users in North America, Europe, and Asia. This is not the first time CrowdStrike has faced a major outage.
According to reports, the outage was caused by a technical issue with CrowdStrike's cloud infrastructure. This is a critical component of the company's services.
The outage lasted for several hours, causing disruptions to businesses and organizations that rely on CrowdStrike's security solutions. The impact of the outage is still being assessed.
Causes and Analysis
The CrowdStrike outage was not caused by a Microsoft Windows flaw, but rather a flaw in CrowdStrike Falcon that triggered the issue.
The flaw was in CrowdStrike Falcon's sensor configuration update, specifically in a file called "channel file 291", which was meant to improve how Falcon evaluates named pipe execution on Microsoft Windows.
A logic error was introduced in channel file 291, causing the Falcon sensor to crash and resulting in Windows system crashes and BSODs.
The error was caused by a mismatch between the number of input fields in the IPC Template Type and the actual inputs provided by the sensor code, which was not caught by CrowdStrike's Content Validator component due to a logic error.
CrowdStrike has since provided a 12-page root cause analysis report explaining the issue in detail, including patches that were made after July 19, 2024, to fix the problem.
Analysis of the
The logic flaw in CrowdStrike Falcon that caused the global outage was not a Microsoft Windows flaw, but rather a flaw in the Falcon sensor itself. This flaw was triggered by a faulty update to the sensor's configuration, specifically channel file 291.
The update was supposed to improve how Falcon evaluates named pipe execution on Microsoft Windows, but it inadvertently introduced a logic error that caused the Falcon sensor to crash. This crash resulted in a Windows system crash and BSOD.
CrowdStrike's Content Validator component failed to catch the error in the update, allowing the faulty version of channel file 291 to pass validation. The company later discovered that the flaw was due to a mismatch between the number of input fields in the IPC Template Type and the actual inputs provided by the sensor code.
The mismatch led to a runtime array bounds check being missing in the Content Interpreter, and the Content Validator contained a logic error. These conditions were both patched by CrowdStrike after July 19, 2024.
A preliminary Post Incident Review (PIR) by CrowdStrike identified a flaw in the Content Validator component, which enabled the faulty version of channel file 291 to pass validation.
Recommended read: Crowdstrike Says Faulty Update Caused Global Outage.
What Is?
CrowdStrike is a U.S. cybersecurity company that provides software to companies worldwide and across various industries.
It bills itself as the globe's most advanced cloud-based security technology provider.
CrowdStrike was founded in 2011 and launched in early 2012.
The company listed on the Nasdaq exchange five years ago.
CrowdStrike reported having 29,000 subscribing customers.
It has a partnership with Amazon Web Services.
Its "Falcon for Defender" security technology is designed to supplement Microsoft Defender to prevent attacks.
Readers also liked: Is Crowdstrike an American Company
Why Apple and Linux Were Unaffected
The July outage that affected Microsoft Windows was a wake-up call for many IT professionals. Apple and Linux systems were not affected because the faulty sensor configuration update, channel file 291, was never issued to these operating systems.
The update dealt specifically with named pipe execution, which only occurs on Microsoft Windows OS, leaving Apple and Linux systems out of the loop. This was a deliberate design choice to limit potential risk.
Apple and Linux systems have different integration points for the Falcon sensor, which is why they were not impacted by the outage. This difference in integration points was a key factor in preventing the outage from spreading to these operating systems.
In fact, Red Hat, a Linux vendor, did experience an incident in June where the Falcon sensor triggered a kernel panic, but it was resolved without any major incidents. This highlights the importance of having different integration points for the Falcon sensor in Linux systems.
Related reading: Was Ups Affected by Crowdstrike
Business Impact
The CrowdStrike outage had a significant impact on businesses worldwide. The recovery process was complex and time-consuming, taking anywhere from a few days to potentially months for some organizations.
CrowdStrike itself was able to identify and deploy a fix for the issue in just 79 minutes, but the recovery process was a different story. IT administrators had to manually boot affected systems into Safe Mode or the Windows Recovery Environment to delete the problematic channel file 291 and restore normal operations.
For some businesses, the process was labor-intensive, especially for those with many affected devices. In some cases, physical access to each machine was required, adding further time and effort to the process.
The use of Microsoft Windows BitLocker encryption technology made it even more time-consuming to recover, as BitLocker recovery keys were required. It was initially estimated that it could take months for some organizations to entirely recover all affected systems from the outage.
By July 29, 2024, CrowdStrike reported that approximately 99% of affected Windows sensors were back online.
Check this out: Is It a Good Time to Buy Crowdstrike Stock
Company Reactions
CrowdStrike CEO George Kurtz apologized for the global outage in a blog post, saying the company is working closely with impacted customers and partners to restore all systems. He encouraged customers to remain vigilant and engage with official CrowdStrike representatives for support.
CrowdStrike's blog and technical support portal will remain the official channels for updates, and Kurtz reiterated that the outage was caused by a defect in a Falcon content update for Windows hosts, not a security breach or cyberattack.
Businesses are still reeling from the outage, with many flights grounded and payments systems downed.
Microsoft CEO Satya Nadella wrote in a statement that an update from CrowdStrike on Thursday impacted global IT systems.
Tesla employees reported that some manufacturing lines were slow to start on Friday morning, and others were temporarily halted in California and Nevada.
Readers also liked: Delta Ceo Says Crowdstrike Outage Will Cost Carrier $500 Million.
System Failures and Errors
The "blue screen of death" error message has been affecting Microsoft users worldwide due to a major outage caused by an update from cybersecurity firm CrowdStrike.
The outage started around 19:00 UTC on July 18th and has been impacting Virtual Machines running Windows Client and Windows Server.
Microsoft has confirmed that the affected update has been pulled by CrowdStrike, and customers experiencing issues should reach out to CrowdStrike for assistance.
Switzerland's National Cyber Security Service (NCSC) has received reports from various companies and critical infrastructures in Switzerland, blaming the outage on a faulty update or misconfiguration by CrowdStrike.
Germany's Allianz has been impacted by the outage, with employees unable to log into their computers, affecting multiple companies besides Allianz.
CrowdStrike has deployed a fix to a defect found in an update for Windows hosts, but a manual fix may be required for some customers.
The fix may not be incredibly straightforward, as suggested by Tom Lysemose Hansen, Chief Technology Officer at Promon, who notes that affected customers will have to break into their own systems to get everything back online.
Cybersecurity experts warn of bad actors who may reach out claiming they can help, so it's essential to only talk to trusted organizations as you work towards recovery.
CrowdStrike is now rolling back the update globally after the major outage, which has been impacting businesses worldwide.
You might enjoy: Crowdstrike Fix Deployed
Financial and Market Impact
The global outage of Crowdstrike had a significant impact on the financial markets. The company's stock price plummeted by 10% in a single day.
The financial losses for Crowdstrike's customers were substantial, with some companies reporting losses in the millions of dollars. This was due to the disruption of their cybersecurity services, which were critical to their operations.
The global nature of the outage meant that many companies were affected, including those in the finance and healthcare sectors. These sectors are heavily reliant on cybersecurity services to protect sensitive data.
The outage also highlighted the importance of having a robust disaster recovery plan in place. This is something that Crowdstrike's customers are now likely to prioritize.
The company's reputation took a hit as a result of the outage, which may impact its ability to attract new customers. This could have long-term financial implications for the company.
Take a look at this: Can Companies Sue Crowdstrike
Government and Regulatory Response
The government is taking steps to address the CrowdStrike outage. The Department of Homeland Security is working with CrowdStrike, Microsoft, and government agencies to evaluate the outage.
The Department of Homeland Security is collaborating with multiple organizations to assess and address the system outages. This includes CrowdStrike, Microsoft, and federal, state, local, and critical infrastructure partners.
The University of Miami's information technology team is actively working to resolve the disruption and bring all systems back online.
Ftc Chair Lina Khan on Competition and Fragile Systems
FTC Chair Lina Khan blames the CrowdStrike outage on concentrated market power, which creates "fragile systems."
Khan argues that concentrating production can concentrate risk, leading to cascading effects from a single disruption.
She didn't announce an FTC investigation of the incident but linked to unrelated past actions, including a query on cloud computing companies and seeking public comment on companies that frequently buy smaller competitors.
Khan's comments suggest that she views the outage as a symptom of a larger issue with concentrated market power.
Expand your knowledge: Crowdstrike Marketcap
Dhs Assessing and Addressing with Others
The Department of Homeland Security is taking proactive steps to address the system outages caused by CrowdStrike's faulty update. They're working with CrowdStrike, Microsoft, and government agencies to evaluate the issue.
The DHS is collaborating with multiple partners to fully assess and address the system outages. This includes the Cybersecurity and Infrastructure Security Agency, which is also part of the effort.
The University of Miami's information technology team is actively working to resolve the disruption and bring all systems back online.
Prevention and Future
CrowdStrike has taken steps to prevent similar incidents in the future. They have changed their content update procedures to make them more robust.
Updates are now treated like code updates, with internal testing and phased implementation. This means that potential issues are caught before they affect customers.
A "system of concentric rings" approach has been implemented for rolling out updates. This allows for a more controlled and gradual rollout.
Customers now have more control over updates, with options to be early adopters, have general availability, or opt-out/delay. This gives them more flexibility and peace of mind.
Business Recovery Timeline
It took CrowdStrike just 79 minutes to identify and deploy a fix for the issue, but recovering from the outage was a complex and time-consuming process for businesses.
CrowdStrike reported that approximately 99% of affected Windows sensors were back online as of July 29, 2024.
IT administrators had to manually boot affected systems into Safe Mode or the Windows Recovery Environment to delete the problematic channel file 291 and restore normal operations.
The process was labor-intensive, especially for organizations with many affected devices, and in some cases, required physical access to each machine.
Businesses with extensive IT infrastructure and encrypted drives were significantly impacted, with the use of Microsoft Windows BitLocker encryption technology making recovery even more time-consuming.
Explore further: Is Apple Affected by Crowdstrike
Preventing Future Incidents
CrowdStrike has taken steps to prevent similar incidents in the future.
They've changed their content update procedures to treat updates like code updates, with internal testing and phased implementation.
This new approach is designed to reduce the risk of outages caused by rapid response content updates.
CrowdStrike has also implemented a "system of concentric rings" approach for rolling out updates, which helps to contain disruptions.
Customers now have more control over updates, with options to choose their level of adoption: early adopter, general availability, or opt-out/delay.
By giving customers more flexibility, CrowdStrike aims to minimize the impact of updates on their operations.
This new approach is a significant improvement over previous methods, which often led to widespread outages.
By learning from past mistakes, CrowdStrike is taking proactive steps to prevent future incidents.
Here's a summary of the new update procedures:
Industry-Specific Impact
CrowdStrike's impact on various industries is a significant topic. The company's solutions have been adopted by numerous sectors, including finance and healthcare.
In the finance sector, CrowdStrike's endpoint security platform has been used to protect against cyberattacks, such as the one that targeted the financial services company, where a phishing attack was launched via a fake email.
Healthcare organizations have also implemented CrowdStrike's solutions to safeguard against data breaches, like the one that exposed sensitive patient information.
Shipping Disruptions: FedEx and UPS
FedEx and UPS are both experiencing significant disruptions due to the global tech outage. FedEx is reporting "substantial disruptions" that could lead to delays in package deliveries.
The shipping company is attempting to mitigate the impact by activating contingency plans to address the disruption throughout its networks. UPS is also reporting service delays, but its airline and driver delivery systems are still operating effectively.
FedEx is expecting delays for package deliveries that were supposed to arrive on Friday. UPS is working to resolve all issues caused by the software outage that's affecting some of its computer systems in the U.S. and Europe.
DHL, on the other hand, says the outage has had only a very limited and local impact, but some suppliers and business partners are still affected.
Related reading: Crowdstrike Financial Impact
Flight Delays
Flight delays were a major issue for travelers. Several airlines requested assistance with ground stops due to the IT outage.
The Federal Aviation Administration confirmed that American Airlines, Delta Air Lines, and United Airlines were among those affected. These airlines were forced to resume flights in a limited capacity.
Delta and United issued travel waivers to allow customers to change their plans. This was a welcome relief for passengers who were already facing delays.
Frequently Asked Questions
Has the CrowdStrike issue been resolved?
CrowdStrike has fixed the issue on its end, but some systems may still experience prolonged recovery times. Further resolution depends on ongoing collaboration with customers.
Featured Images: pexels.com
