The attackers who pulled off the recent breach of continuous integration and continuous delivery (CI/CD) platform maker CircleCI got in by compromising an engineer’s laptop with malware, stealing their 2FA-backed SSO session cookie, and using it to impersonate the employee in a remote location.
“Because the targeted employee had privileges to generate production access tokens as part of the employee’s regular duties, the unauthorized third party was able to access and exfiltrate data from a subset of databases and stores, including customer environment variables, tokens, and keys,” CircleCI CTO Ron Zuber explained.
“Though all the data exfiltrated was encrypted at rest, the third party extracted encryption keys from a running process, enabling them to potentially access the encrypted data.”
The timeline of the CircleCI breach
Earlier this month, when the company revealed that they’ve been breached, it urged customers to “rotate any secrets stored in CircleCI.”
In the following days, the company continued to take actions to minimize the damage customers could experience due to this breach, but confirmed on Friday that fewer than 5 customers have informed them of unauthorized access to third-party systems as a result of this incident.
The attackers had plenty of time to do damage. According to Zuber:
- The engineer’s laptop was compromised on December 16, 2022
- The unauthorized third-party access to CircleCI systems occurred on December 19
- Exfiltration of data occurred on December 22
The malware on the engineer’s laptop was not detected by the company’s antivirus software, and attackers’ impersonation of the employee also went unnoticed.
It was only on December 29, when they were alerted to suspicious GitHub OAuth activity by one of their customers, that they began to investigate and unearthed evidence of compromise.
Mitigation and remediation
In the following week, they shut down all access for the employee whose account was compromised and shut down production access to most of the rest, then proceeded to:
- Rotate potentially exposed production hosts
- Revoke Project API and Personal API tokens
- Rotate GitHub OAuth tokens
- Work with Atlassian to rotate all Bitbucket tokens on behalf of customers
- Work with AWS to notify customers that their AWS tokens could have been compromised
They have now also shared indicators for compromise to help customers with their own investigations. “We recommend you investigate for suspicious activity in your system starting on December 16, 2022 and ending on the date you completed your secrets rotation after our disclosure on January 4, 2023. Anything entered into the system after January 5, 2023 can be considered secure,” Zuber noted.
He also laid out the additional defensive layers they have put into place to prevent future attacks of this kind. “We want to be clear. While one employee’s laptop was exploited through this sophisticated attack, a security incident is a systems failure. Our responsibility as an organization is to build layers of safeguards that protect against all attack vectors,” he concluded.