Thousands of applications that have taken advantage of open source Python Package Index (PyPI) software packages may at risk of hijacking and subversion by malicious actors, opening up the possibility of major supply chain attacks affecting even greater numbers of downstream organisations and users.
This is according to threat researchers at jFrog, who identified the technique being exploited in the wild against the pingdomv3 package – part of the widely used Pingdom API website monitoring service, owned by SolarWinds – while monitoring the open source ecosystem. The team has dubbed the technique Revival Hijacking.
The technique itself is similar in its fundamentals to typosquatting – where threat actors take advantage of common spelling errors, to register malicious domains.
In the Revival Hijack attack against the pingdomV3 package, an undisclosed threat actor took advantage of a PyPl feature whereby when a package is deleted or removed from the repository, its name becomes immediately available for use again.
As the name suggests, this means the package can effectively be revived and hijacked for nefarious purposes.
JFrog’s Brian Moussali, malware research team leader, who co-authored the resulting report, said the Revival Hijack technique was particularly dangerous for three main reasons.
Firstly, unlike typosquatting, the technique does not rely on its victim making a mistake when installing the malicious package. Secondly, updating a known safe package to its latest version is a common practice that many developers see as minimal in its risk – even though that is not the case. Thirdly, many CI/CD machines will be set up to install package updates automatically.
“The Revival Hijack is not just a theoretical attack – our research team has already seen it exploited in the wild. Using a vulnerable behaviour in the handling of removed packages allowed attackers to hijack existing packages, making it possible to install it to the target systems without any changes to the user’s workflow,” said Moussali.
“The PyPI package attack surface is continually growing. Despite proactive intervention here, users should always stay vigilant and take the necessary precautions to protect themselves and the PyPI community from this hijack technique.”
Moussali and his co-researcher Andrey Polkovnichenko say that based on a back-of-a-napkin count of removed PyPI packages, as many as 120,000 could potentially be hijacked. Filtering out those that have under 100,000 downloads, have not been active long, or that are obviously dodgy, the figure still tops 22,000.
And with an average of 309 PyPI projects being removed every month, anybody keen to exploit the Revival Hijack technique has a steady stream of potential new victims.
What happened to pingdomV3?
In the case of pingdomV3, the original owner of the package, who appears to have moved on, last updated it in April 2020, then went quiet until 27 March 2024 when they sent a brief update telling users to avoid using the package as it was abandoned. They then removed it on 30 March, at which point the name popped up for registration.
Almost immediately, a user with a Gmail address published a package under the same name with a newer version number, claiming it to be a redevelopment and pointing it to a GitHub repository. This version contained the standard pingdomV3 code, although the linked GitHub repository actually never existed.
Then, on 12 April, jFrog’s automated scanners detected weird activity when the owner introduced a suspicious, Base64-obfuscated payload. This set alarm bells ringing and prompted the investigation and subsequent disclosure. The package was removed altogether by PyPI on 12 April, and its name has been prohibited from use.
The payload itself appeared to be a Python trojan malware designed to discover if it is running in a Jenkins CI setting, in which case it performs an HTTP GET request to an attacker-controlled URL. The JFrog team was not able to retrieve the ultimate payload that this would have delivered, which suggests the malicious actor either wanted to delay their attack, or was limiting it to a specific IP range. In any case, it was thwarted.
Concerned at the potential scope of the problem, Moussali and Polkovnichenko then set about hijacking the most downloaded abandoned packages themselves, and replacing them with empty, benign ones, all with version number 0.0.0.1 to make sure they weren’t accidentally pulled in automated updates.
Checking back after a few days, they found that their empty PyPI packages had been downloaded over 200,000 times.
Of course, since the replacement packages are empty it is not possible to say with much confidence that a malicious actor could actually have achieved code execution every time, but “it would be very safe to say” that in the majority of cases they would, said Moussali.
PyPI’s response
According to jFrog, PyPI has been considering a policy change on deleted packages that would eliminate this loophole, but for some reason no conclusion on this has been reached in over two years of deliberation.
It does make it clear, on deletion, that the name will be released for use to others, and it does also prevent specific versions of packages from being deleted, in line with OpenSSF recommendations.
However, said Moussali, while this is helpful, the potential scope of the Revival Hijack technique is so extensive that more action is needed.
“We fully advocate PyPI to adopt a stricter policy which completely disallows a package name from being reused. In addition, PyPI users need to be aware of this potential attack vector when considering upgrading to a new package version,” he wrote.
Henrik Plate, a security researcher at Endor Labs, said: “This risk is real, and depends on the popularity of the package. The risk probably decreases if packages have been deleted a long time ago, because the longer a package has been taken down, the more developers and pipelines have noticed its unavailability and adapted their dependency declarations.
“In this context, it is noteworthy that the example provided was revived just shortly after the deletion, which could indicate that the attacker monitored package deletions on PyPI.
“Reviving deleted packages is a known problem. The taxonomy of supply chain attack vectors visualised by the Endor Labs Risk Explorer (a fork of the GitHub project sap/risk-explorer) covers this vector as [AV-501] Dangling Reference, and supporting examples include revived GitHub repositories, renamed GitHub repositories and revived npm packages,” Plate told Computer Weekly in emailed comments.
Plate went on to state that this underlines the importance of stricter security guidelines for package repositories, such as those suggested by OpenSSF.
For defenders, he said, using internal package registries should protect developers from such attacks by mirroring open source packages such that they remain available even if deleted. However, cautioned Plate, such internal registries do need to be configured so that new, potentially malicious package versions are thoroughly vetted prior to mirroring.