CircleCI is facing customer ire and possible regulatory enquiry after it disclosed a security incident and asked its customers to rotate their secrets. However, CircleCI customers must immediately start proactive defence measures, warned Or Aspir, Principal Security Researcher and Developer at Mitiga.
The San Francisco-based business – one of the largest providers of continuous integration and continuous delivery services – issued a warning to its customers on January 4 to rotate all their secrets due to a “security incident”.
The company stated that an investigation is currently in progress, but reassured customers that “there are no unauthorized actors active in our systems.”
It urged its customers to review internal logs for signs of unauthorized access between December 21, 2022, and January 4, 2023, or until the secrets are rotated. It updated the advisory on January 7, saying that it completed rotating GitHub OAuth tokens on behalf of customers.
However, instead of merely rotating the data, customers should hunt for malicious actions on all their integrated SaaS and cloud platforms to make sure that there were no breaches on the connected platforms such as GitHub or AWS, Mitiga’s Or Aspir told The Cyber Express.
CircleCI and secrets in code
Keeping secrets in code is a well-known security hazard, but with the advancements in code-driven automation and CI/CD, it’s a practice that is still widely used. Unfortunately, this can have serious consequences, as demonstrated by a recent security incident at CircleCI.
The exact details of the incident are still unclear, but CircleCI CTO Rob Zuber has advised customers to take immediate action. This includes rotating any secrets stored in CircleCI, reviewing internal logs for unauthorized access, and replacing any Project API tokens.
Despite the incident, Zuber assures customers that they can continue to build their projects on CircleCI’s service. This exactly is the root of the trouble, Aspir told The Cyber Express.
Users of CircleCI platform integrate it with other SaaS and Cloud providers they use, such as GitHub, Jira, Kubernetes, and AWS. Each integration happens when the customer provides the CircleCI platform with authentication tokens and secrets.
This also means that a threat actor who accesses CircleCI can access the secrets that are stored within the platform. As such, a security incident involving the client’s CircleCI platform puts all its connected SaaS platforms and cloud providers at risk.
Need to hunt CircleCI loops
So, is the threat big enough? “Any threat you can think of,” Aspir told The Cyber Express.
“Let’s assume you saved the credentials of AWS user in the CircleCI platform. The hacker can steal the credentials and login to that user and do whatever the permissions provided by that user, like stealing data, running crypto mining on compute resources and more,” he explained.
“Another example of potential threat is if the CircleCI platform is integrated with the company Git repository. The attacker could steal company code, if they stole the Git credentials saved in the CircleCI platform.”
According to the Mitiga threat hunting guide, the details CircleCI security team shared are incomplete. It recommended tracing suspicious activity in the CircleCI audit logs by keeping these points in mind.
CircleCI’s audit logs provides information about actions taken, by which actor, on which target, and at what time. The factors to note are
The action taken that created the event: The format is ASCII lowercase words, separated by dots, with the entity acted upon first and the action taken last. In some cases, entities are nested, for example, workflow.job.start.
The actor who performed this event: In most cases this will be a CircleCI user. This data is a JSON blob that will always contain id and type and will likely contain name.
The target, or the entity instance acted upon for this event: A project, an org, an account, or a build, for example. This data is a JSON blob that will always contain id and type and will likely contain name.
The payload: A JSON blob of action-specific information. The schema of the payload is expected to be consistent for all events with the same action and version.
The time: When the event occurred in UTC is expressed in the ISO-8601 format, with up to nine digits of fractional precision. For example: ‘2017-12-21T13:50:54.474Z’.
“Security personnel may think that by rotating all the secrets or tokens they have done with the incident. But we need to remember that the incident timeline is from December 21, 2022, through January 4, 2023,” said Aspir.
“So, in this time a hacker could authenticate to other SaaS platforms that are configured with the CircleCI platform and gain access or download data and other malicious actions. So, you need to go back and do a hunt on all those SaaS providers.”
CircleCI and a series of unfortunate events
In November, CircleCI had warned about attempted phishing attacks against organizations by attackers pretending to be from the company.
“None of CircleCI’s systems have been compromised, and our customers’ data and information remain safe,” wrote company CTO Rob Zuber then.
“The privacy and security of our customers’ data is paramount for CircleCI, and we want to remind you to stay vigilant and aware of attempts to gain access to code repositories.”
In December 2022, the company announced terminating 17% of its staff from employment. “When something of this magnitude happens, it’s not business as usual,” CEO Jim Rose wrote then. The words appear prophetic now.