CrowdStrike blames the crash on a buggy security content update

A recent buggy “security content configuration update” to CrowdStrike’s Falcon sensor, designed for gathering telemetry on new threat techniques for Windows, has been confirmed as the root cause of a major global incident on July 19. This update crash has left IT teams worldwide scrambling for solutions, according to CrowdStrike.

Incident Overview

CrowdStrike, a leading provider of endpoint detection and response (EDR) solutions, serves around 29,000 customers with its cloud-based software. On July 19, a defective Rapid Response Content configuration update caused a global disruption, affecting business continuity, travelers, hospital patients, and professionals.

CrowdStrike released a “preliminary Post Incident Review (PIR)” detailing the incident. The report identified a defect in Channel File 291’s content as the cause. When the problematic content was loaded into the Content Interpreter, it triggered an out-of-bounds memory read, leading to a Windows operating system crash (BSOD).

Rapid Response Content and the Falcon Platform

CrowdStrike uses Rapid Response Content to update its software with the latest threat intelligence. This process is crucial for maintaining the dynamic protection mechanisms of the Falcon platform. These updates enable behavioral pattern-matching operations on the Falcon sensor by delivering new security content.

Technical Breakdown of the Incident

The sensor involved in the incident, sensor 7.11, was deployed on February 28 and introduced a new IPC Template Type to detect novel attack techniques abusing Named Pipes. After passing a stress test on March 5, it was released to production.

Between April 8 and April 24, three additional IPC Template Instances were deployed and performed as expected. However, on July 19, two new IPC Template Instances were deployed, one of which contained problematic content data. A bug in the Content Validator allowed this faulty content to pass validation, resulting in the global Windows crash.

CrowdStrike’s Response and Future Mitigation Plans

CrowdStrike’s CEO, George Kurtz, has been called to testify before Congress about the incident, underscoring its severity. To restore its reputation and prevent future issues, CrowdStrike has outlined several measures:

1. Enhanced Testing Procedures:
   • Local developer testing
   • Content update and rollback checks
   • Stress tests and fuzzing
   • Fault injection, stability, and content interface testing
2. Improved Content Validation:
   • New validation checks to prevent problematic content deployment
   • Enhanced error handling in the Content Interpreter
3. Staggered Deployment Process:
   • Gradual deployment to larger portions of the sensor base, starting with a canary deployment
   • Improved monitoring for sensor and system performance to guide phased rollouts
4. Customer Control and Communication:
   • Greater customer control over content update delivery
   • Granular selection of when and where updates are deployed
   • Detailed release notes via subscription

Industry Impact and Observations: David Ferbrache, managing director of Beyond Blue, a cybersecurity consulting firm, notes that CrowdStrike, previously known mainly within security and technology circles, is now widely recognized due to this incident. The company’s challenge is to balance the risk of rapid updates with the need for security, allowing customers to manage their update schedules to mitigate potential risks.

CrowdStrike’s proactive steps to improve its update process and communication with customers aim to prevent similar incidents in the future, ensuring a more robust and reliable endpoint security solution.

By addressing these issues and implementing comprehensive testing and validation procedures, CrowdStrike seeks to maintain its position as a leader in the cybersecurity industry while providing its customers with the confidence and control needed to manage their security effectively.

4o