Amazon Web Services (AWS) has introduced the same remote code execution (RCE) vulnerability three times over the last four years through its Neuron SDK, highlighting critical lapses in securing its Python package installation processes.
Despite previous warnings and fixes, the same dependency confusion vulnerability has resurfaced with new package releases in its software ecosystem.
The issue was first discovered in April 2022 when Giraffe Security flagged a vulnerability in AWS’s Neuron SDK, a set of Python libraries enabling machine learning workloads on AWS’s specialized hardware.
The problem stemmed from AWS’s official installation instructions and documentation, which recommended a command like the following:
pip install transformers-neuronx --extra-index-url=https://pip.repos.neuron.amazonaws.comAt a glance, the command seems simple, instructing Python’s pip package manager to install the package transformers-neuronx from the AWS-specific repository (https://pip.repos.neuron.amazonaws.com). However, this approach contains a hidden danger rooted in how pip handles the parameters.
The Technical Issue
The --extra-index-url parameter does not exclusively restrict package downloads to the specified private repository.
Instead, it allows pip to search the default public PyPi repository for packages, falling back on it if the package is not found in the specified index. This creates a critical vulnerability: malicious actors could upload a package with the same name to PyPi, tricking users into downloading and executing malicious code.
In 2022, Giraffe Security confirmed this vulnerability by claiming unprotected AWS package names like mx-neuron on PyPi and reporting the flaw through AWS’s bug bounty program.
AWS promptly addressed the issue by uploading placeholder “dummy” versions of the affected packages to PyPi, preventing further exploitation. However, the root cause—a flawed reliance on the --extra-index-url parameter—remained unaddressed.
Investigate Real-World Malicious Links, Malware & Phishing Attacks With ANY.RUN – Try for Free
A Pattern of Neglect: 2020 and 2022
Further research in 2022 revealed that this was not the first instance of such a vulnerability. Historical data from libraries.io, a database of open-source software, showed that AWS’s torch-neuron package had been similarly exposed in 2020, demonstrating an earlier instance of the same dependency confusion risk.
At the time, a security researcher had uploaded numerous versions of the package to PyPi to highlight the flaw, forcing AWS to take corrective action.
Despite being aware of the issue since at least 2020, AWS failed to implement a lasting solution, leading to repeated vulnerabilities being exposed in 2022.
Hat Trick: The 2024 Discovery
Despite multiple warnings and fixes over the years, Giraffe Security’s latest investigation in December 2024 revealed that AWS had once again introduced the same vulnerability.
The Neuron SDK’s private package index had expanded significantly, but AWS had neglected to preemptively claim the newly added package names on PyPi. This allowed Giraffe Security to successfully register some of the new package names under its own PyPi account—a clear indication that AWS failed to learn from past mistakes.
Amazon’s repeated missteps raise questions about their approach to addressing this issue. On one hand, their quick response to past reports suggests that they take the vulnerability seriously. However, the recurrence of the same flaw indicates a lack of systemic processes to prevent it.
There could be different perspectives driving this oversight:
- A Documentation Misstep: AWS might view this as a configuration issue caused by customers improperly interpreting their documentation. While experienced Python developers are aware of the risks associated with
--extra-index-url, AWS’s reliance on this parameter in official tutorials is arguably misleading. - A Low Priority Issue: AWS may not consider the problem critical, believing that customers bear responsibility for ensuring the security of their installations. However, this stance ignores the fact that AWS’s official documentation directly contributes to the issue.
This situation underscores a critical security lesson: even trusted sources like official AWS documentation are not immune to mistakes.
Developers should always scrutinize and fully understand package installation processes before implementing them in production systems. Safer alternatives—such as using the --index-url parameter to restrict downloads exclusively to private repositories or leveraging modern package managers like Poetry—should be considered.
While this recurring issue may seem like a niche vulnerability, it has broader implications for security in the cloud ecosystem.
Dependency confusion attacks have become a growing concern, particularly as more organizations rely on private package registries in tandem with public repositories like PyPi or npm.
The responsibility to mitigate these risks lies not only with end-users but also with service providers like AWS, who must ensure their tools and documentation follow security best practices.
AWS’s Silence
Despite repeated attempts to contact Amazon for comment, Giraffe Security has not received a response. As one of the largest cloud providers globally, AWS’s lack of a robust and permanent solution in this case is surprising, especially given its reputation for prioritizing security.
AWS’s repeated failures to address the same RCE vulnerability in its Neuron SDK highlight a concerning lapse in security processes. While the immediate fixes to individual reports have been swift, the lack of a permanent resolution to the underlying issue remains troubling.
Find this News Interesting! Follow us on Google News, LinkedIn, and X to Get Instant Updates!