CrowdStrike blames global outage on buggy update

Company vows to do better with improvements to testing and deployment

Pro

CrowdStrike has released a statement outlining the causes of last Friday’s incident that saw millions of Windows systems around the world crash. Mac and Linux hosts were not impacted.

The company laid the blame for the incident not on hacktivists or cyber criminals but on something more mundane: a defective content configuration update.

CrowdStrike delivers security content configuration updates sensors in two ways: through ‘sensor content’ that is shipped with a sensor directly, and ‘rapid response content’ that is flexible and designed to respond to the threats at speed. The issue on Friday involved a Rapid Response Content update with an undetected error.

Rapid response content is used to perform behavioural pattern-matching operations on the sensor and is a representation of fields and values, with associated filtering. It is stored in a proprietary binary file that contains configuration data delivered as ‘template instances’ which map to specific behaviours for the sensor to observe, detect or prevent. Template instances have a set of fields that can be configured to match the desired behaviour.

Rapid response content provides visibility and detections on the sensor without requiring sensor code changes. This capability is used by threat detection engineers to gather telemetry, identify indicators of adversary behaviour and perform detections and preventions. It is based on behavioural heuristics, separate and distinct from CrowdStrike’s on-sensor AI prevention and detection capabilities.

Last Friday, two additional IPC template instances were deployed. Due to a bug in the content validator, one of the instances passed validation despite containing what the company called “problematic content data” that led to the downing of some 8.5 million Microsoft systems.

In its statement CrowdStrike outlined a number of measures to prevent a recurrence, including better testing of rapid response content and revised deployment giving greater control of the delivery of updates.

CrowdStrike added that is would be releasing a comprehensive analysis once an internal investigation was complete.

George Kurtz, CrowdStrike Founder and CEO, said: “The outage was caused by a defect found in a Falcon content update for Windows hosts. Mac and Linux hosts are not impacted. This was not a cyberattack.

“We are working closely with impacted customers and partners to ensure that all systems are restored, so you can deliver the services your customers rely on.

“CrowdStrike is operating normally, and this issue does not affect our Falcon platform systems. There is no impact to any protection if the Falcon sensor is installed. Falcon Complete and Falcon OverWatch services are not disrupted.”

Read More: CrowdStrike cyber security security