Attackers are leveraging a vulnerability (CVE-2023-48022) in Anyscale’s Ray AI software to compromise enterprise servers and saddle them with cryptominers and reverse shells.
“To our knowledge, the attack started 7 months ago,” Avi Lumelsky, a researcher at Oligo Security, told Help Net Security.
“We observed hundreds of compromised clusters in the past three weeks alone. Each cluster uses a public IP address, and most clusters contain hundreds to thousands of servers. There are hundreds of servers that are still vulnerable and exposed.”
About the vulnerability (CVE-2023-48022)
The open-source Ray framework is used for scaling AI and Python applications from a laptop to a cluster and to accelerate machine learning (ML) workloads.
“According to Anyscale, some of the world’s largest organizations use Ray in production, including Uber, Amazon, and OpenAI,” Oligo researchers noted.
CVE-2023-48022 was privately flagged by Bishop Fox researchers in August 2023, and it stems from the fact that the framework does not not enforce authentication before allowing access to its Jobs API.
Following the disclosure, Anyscale has decided not to ship a fix (i.e., add authentication to the Ray dashboard), and contended that Ray’s interface should not be exposed on the internet and should only be accessible to trusted parties, and that the lack of authentication requirements is a feature, not a bug.
But, they added, “while we still do not believe that an organization should rely on isolation controls within Ray like authentication, there can be value in certain contexts in furtherance of a defense-in-depth strategy, and so we will implement this as a new feature in a future release.”
In the meantime, though, organizations have been exposing their Ray network services to the internet, and the opportunity has been seized by attackers.
Attackers can do a lot of damage
“The first crypto-miner we noticed was installed on Feb. 21, 2024. Using public web intelligence tools, we discovered that the IP has been accepting connections to the target port since Sept. 5, 2023, indicating the breach might have started before the vulnerability was disclosed,” Oligo Security researchers noted.
“Due to the scale of the attacks and the chain of events, we believe the threat actors are probably part of a well-established hacking group.”
But the attackers did not just use the Ray clusters for covert cryptomining – they also installed reverse shells, to establish a permanent connection with the servers and allow them to control them remotely.
“We found numerous sessions that are open to external IP addresses and we found traces in the Jobs API showing exactly how the attackers did what they did,” Lumelsky shared.
The compromised machines included a wealth of sensitive information, including:
- OpenAI tokens (which can be used to access OpenAI accounts)
- Stripe tokens (which can be used to drain Stripe payment accounts)
- HuggingFace tokens (which may allow attackers to access private repositories and fiddle with ML models)
- Slack tokens (attackers may use to read an affected organization’s Slack messages or send messages)
- Production DB credentials (allowing attackers to download/modify databases), and more.
“AI production workloads were compromised, meaning an attacker could affect an AI model’s integrity or accuracy, steal models, and infect models during the training phase,” the researchers added.
“Impacted organizations came from many industries, including medical companies, video analytics companies, elite educational institutions, and many more.”
Clean-up and prevention
Lumelsky pointed out that it’s not possible to know exactly what actions the attackers may have done with this sensitive information.
“We know that truly, crypto mining is one of the better-case scenarios. If the attackers had chosen instead to create malicious models and alter the output of AI being used in sensitive applications, the impact could be enormous,” he commented.
Oligo researchers have helped multiple companies mitigate unauthorized access to their clusters, and those clusters are no longer exposed. They have also shared indicators of compromise and offered advice on mitigation strategies to implement.
Help Net Security has reached out to Anyscale, and has been told that the company is “currently working on a script that will make it easy for users to verify their configuration and avoid accidental exposure.”