If you’re managing servers you may need to cancel your weekend plans as a CrowdStrike update has caused servers to BSOD / boot loop.
The incident does not appear to be a security incident or cyberattack, and only affects Windows hosts, with CrowdStrike saying Linux and Mac are not affected.
The issue was first reported 19:00 UTC on July 18 and was acknowledged by CrowdStrike in the early hours of July 19.
“CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts,” CrowdStrike CEO George Kurtz wrote on Twitter/X.
“This is not a security incident or cyberattack,” he added, “the issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.”
The good news is that a fix has already been found. The bad news is that as servers are not booting it is likely many will require manual intervention. CrowdStrike gave the following instructions on how to fix the issue.
- Boot Windows into Safe Mode or the Windows Recovery Environment
- Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
- Locate the file matching C-00000291*.sys* and delete it
- Boot the host normally
Microsoft later issued further advice:
- We recommend customers that are able to, to restore from a backup from before 19:00 UTC on the 18th of July
- Alternatively, attempt to repair the OS disk offline.
- Attach a disk to VM for offline repair (Encrypted disks may need further instructions)
- Once the disk is attached delete the Windows/System/System32/Drivers/CrowdStrike/C00000291*.sys file
- We can confirm the affected update has been pulled by CrowdStrike. Customers that are continuing to experience issues should reach out to CrowdStrike for additional assistance.
Who is affected by the CloudStrike update?
The CrowdStrike update has affected Windows devices and Virtual Machines running Windows Client and Windows Servers running the CrowdStrike Falcon agent. Personal PCs running Windows are not affected.
It’s not yet known exactly how many machines have been affected but it’s already had a large impact on the globe especially in Europe with Visa, Amazon, and Microsoft all reporting issues. There have also been reports of airlines and hospitals having issues. We won’t know the full extent of the impact until later in the day.
How to fix the CrowdStrike issue?
Essentially, you need to delete the file matching C-00000291*.sys
You can do that by
1. Boot Windows into Safe Mode or the Windows Recovery Environment
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching C-00000291*.sys and delete it
or
You may need to manually remove /update the OS disk
What is CrowdStrike?
CrowdStrike is a cybersecurity company behind software used by some of the largest companies and institutions around the world, including hospitals, airports, banks, and many businesses listed in the Fortune 500.
