CrowdStrike is actively working to resolve a defect in a content update that struck about 8.5 million Windows machines on July 19 – and continues to disrupt many Windows hosts days later.
In a recent update, the cybersecurity company said it has “tested a new technique to accelerate impacted system remediation. We’re in the process of operationalizing an opt-in to this technique. Customers are encouraged to follow the Tech Alerts for latest updates as they happen and they will be notified when action is needed.”
Microsoft has also released a fix for the faulty CrowdStrike update, which resulted in bugcheck and “blue screen of death” (BSOD) errors on millions of Windows hosts.
Delta Airlines was one noteworthy company struggling to recover from the outages, and was still canceling about 20% of its flights as of early afternoon Eastern U.S. time on Monday, July 22. CrowdStrike shares (CRWD) have plunged more than 20% since the incident, erasing roughly $15 billion in market cap.
CEO George Kurtz has assured customers that the faulty update was not due to a cyberattack and that Falcon platform systems remain unaffected.
CrowdStrike Outage Response and Customer Support
The defective update stemmed from a Windows sensor-related content deployment, specifically a channel file in the CrowdStrike directory, which has sparked widespread discussion in the cybersecurity industry about how to ensure that software updates and rollouts are safer and more reliable.
CrowdStrike CSO Shawn Henry took to LinkedIn to apologize for the incident:
“On Friday, though, we failed. The past two days have been the most challenging 48 hours for me over 12+ years. The confidence we built in drips over the years was lost in buckets within hours, and it was a gut punch.
But this pales in comparison to the pain we’ve caused our customers and our partners. We let down the very people we committed to protect, and to say we’re devastated is a huge understatement.
I, and the entire company, take that personally. Thousands of our team members have been working 24/7 to get our customer systems fully restored. The days have been long and the nights have been short, and that will continue for the immediate future. But that is part of the promise we made to all of you when you put your trust and protection in our hands.”
The company quickly mobilized its resources to assist affected customers. A new technique to accelerate system remediation was tested in collaboration with clients, with an opt-in process being implemented. CrowdStrike is providing regular updates through its support portal and social media channels, urging customers to verify communication with official representatives.
Kurtz emphasized the company’s commitment to transparency and customer trust. “Nothing is more important to me than the trust and confidence that our customers and partners have put into CrowdStrike,” he stated. The CEO promised full disclosure on the incident’s cause and preventive measures for the future.
Technical Details and Remediation Steps
For systems still experiencing crashes, CrowdStrike recommends rebooting to download the reverted channel file – multiple times, if necessary. If issues persist, manual or automated remediation options are available, including the use of a bootable USB key for automated fixes.
In response to the widespread issues caused by the faulty update of the CrowdStrike Falcon agent on Windows-based clients and servers, Microsoft released its own recovery tool to help system administrators and IT staff. The updated Microsoft recovery tool offers two repair options – Recover from WinPE (Windows Preinstallation Environment) or Recover from Safe Mode – and also includes guidance for recovering BitLocker encryption keys, if necessary.
As the situation evolves, CrowdStrike continues to prioritize customer support and system restoration, even as the issue of who will pay for the restoration efforts remains unresolved. The company acknowledges the impact of the incident and says it is working tirelessly to regain customer confidence through transparent communication and effective problem-solving.
Shawn stated in his post, “I know I speak for the women and men of CrowdStrike when I say thank you to every customer and partner who has also been working around the clock. You are the real heroes in all of this. We are committed to re-earning your trust by delivering the protection you need to disrupt the adversaries targeting you. Despite this setback, the mission endures.”