Perhaps the defining quality of the software supply chain is complexity. Amid the countless lines of code that the modern world runs on there is potentially infinite scope for mistakes, vulnerabilities and malicious manipulation.
The nature of software development also means that code and tools are constantly being re-used, which in turn are being used to build other applications. From there, the vulnerabilities that might be embedded within one application or code repository – spread quickly out to everywhere else it is used.
In this complex, fast moving supply chain – security debt builds up quickly. Bugs, problems and vulnerabilities get embedded deeply within the software that finally comes to market. From there all it takes is a failure in the right place, or a particularly capable adversary to bring about catastrophe.
Pressures on software development
Our entire world runs on software. That has only become more apparent in recent years and demands for new applications, tools, products and services have exploded. That is reflected in the growing demand for software developers for whose world population is expected to reach 28.7 million by the end of this year, growing by 3.2 million over the last four years.
That said, problems scale along with everything else. That explosion in demand has resulted in a massive increase of pressure on software developers. They’re being asked to develop quicker, do more and release to ever tighter deadlines. Of course, this comes with huge potential for deleterious effects on the final quality of the product.
Our 2022 Fall AppSec Indicator revealed that nearly half – 45% – of developers will in fact skip crucial security steps in order to make those ever tightening deadlines. It’s not hard to see why – 80% of those developers agreed that even those crucial security processes delay delivery. The AppSec indicator found that 74% of respondents admitted to regularly releasing insecure applications. On top of that, 1 in 3 issues under remediation apparently make it to production without being caught in testing or development.
Furthermore, the sheer amount of code being pushed through to production puts pressure on the code review process. This is a stage which requires meticulous concentration and focus, and the specialists who conduct it can and do suffer from overwork and burnout. These pressures can become risky in a single application or service, but they can also spring up at any time throughout the software supply chain as one release goes out to customers or as other developers build upon it. As those releases get passed onto the next link in the chain, so do the errors and bugs that come with them.
That’s just what can happen in a single link, but if we zoom out to the buzzing morass of actors in this supply chain, it’s almost impossible to miss the glaring structural problems too.
Open Source
Make no mistake, modern software development relies on the communal philosophy of Open Source. This is a design philosophy in which people make their code publicly available – thus allowing anyone to use, change and distribute that software. This has become a bedrock resource for software developers in both open-source and private sectors.
The numbers bear it out too. A report from the 2024 Open Source Security and Risk Analysis Report found that open source components are nearly everywhere. Literally. The report found that 96% of all the codebases it reviewed contained open source components. It’s not just that it’s found in nearly all applications – there’s a lot of it too. The report adds that over three quarters – 77% – of the code in those reviewed codebases was open source. It goes further to reveal that “every industry codebase scanned contained open source – most at percentages for 99% to 100%.”
Yet as well-meaning as the philosophy of open source might be – its openness allows for all kinds of errors and the trust that software developers place in it makes those errors particularly dangerous.
That danger emanates from two areas – both perfectly innocent as well as malicious. The first simple point is that the sheer scale of open source components used all along the software supply chain opens up huge scope for vulnerabilities. In fact, given the exponential growth of software, that scope is expanding.
In fact, new vulnerabilities in this huge category open up every single day. A quick Google search will reveal new vulnerabilities cropping up in open source tools almost every day. Recently, for example, security researchers found four vulnerabilities in the widely popular GOGs Git Service, with three of them rated as critical severity.
Of course, vulnerabilities here make their way into the hands of consumers and businesses regularly. Security researchers at EVAsec recently discovered three vulnerabilities – one of which was a decade old – in CocoaPods, an open source tool used to incorporate software libraries into existing applications. CocoaPods – the researchers added – can be found in over three million applications: “Such an attack on the mobile app ecosystem could infect almost every Apple device, leaving thousands of organizations vulnerable to catastrophic financial and reputational damage.”
These were quickly patched by CocoaPods, but only because they were discovered by security researchers – tellingly known as “ethical hackers” – first. Had they been discovered by a malicious party – then the outcome could have been destructive for users of Apple products everywhere.
Then again, attackers know this and are constantly trying to abuse and corrupt open source components to get a foothold into that shared stream of resources that eventually make their way into every sector. Introducing a vulnerability in one of these components, could provide a vulnerability everywhere else it is used.
In fact, attacks on the open source supply chain have skyrocketed in recent years. In 2023, Sonatype revealed that they had seen over 245 thousand attacks against the open source supply chain, showing a 280 percent growth from the previous year.
One particularly destructive example of this is happening right now – Polyfill.io is an enormously popular javascript CDN which thousands of other websites use to nullify the differences that emerge from different versions of a given browser. After a new firm took over the domain early in 2024, the Polyfill.io CDN started delivering malicious javascript to the over 100,000 websites that have embedded cdn.polyfill.io which include jstor and the World Economic Forum.
Supply chain invasion
Of course, attackers don’t actually need to abuse the baseline trust of open source components in order to infect the supply chain and multiply the effectiveness of a given attack. In fact, a 2024 survey from Enterprise Strategy Group has found that 91% of organizations had experienced a software supply chain incident in the previous 12 months. The top vector for those incidents was zero-day exploits from vulnerabilities within third party code.
Software companies that provide widely used applications are also a major target. In the 2020 SUNBURST attacks, attackers inserted malicious code into the update mechanic of Orion – Solarwinds’ flagship infrastructure monitoring and management platform – potentially spreading that malicious code to all the customers who updated – including international businesses, governments and many more. Thankfully, that attack was stopped but only months after it attackers had initially made the initial compromise.
Artificial Intelligence
Looking ahead, Artificial Intelligence may be a game-changer – for good or ill.
Even basic publicly available LLMs – like ChatGPT – have become indispensable tools for software developers. Given the above-mentioned pressures, these tools are now allowing developers to write code even faster. These tools, however, are not infallible and there are plenty of recorded cases of them introducing bugs into the code they generate. Yet another risk emerges when we consider how the ability of these tools to scale code output, will likely result in a scaling of those vulnerabilities within that code.
In fact, a recent study from Stanford University has actually shown that code written without AI-assistance was generally more secure. Those study participants that did use AI-tools to help them write code, turned up significantly more vulnerabilities. Crucially, however, those that used those AI tools imagined that the code they had written was actually more secure than their counterparts. The authors of the paper note – “participants who had access to the AI assistant were more likely to introduce security vulnerabilities for the majority of programming tasks, yet were also more likely to rate their insecure answers as secure compared to those in our control group.”
Feeling dizzy yet?
It’s a headache-inducing amount of complexity to deal with, especially when we consider that these are the supply chains on which we all rely to create and use safe software. In some sense the problem boils down to how hard it is to actually see into these long and complex supply chains. In fact, these are invisible to most. A 2024 survey from Cycode revealed that 72% of IT pros labeled software supply chain security as their biggest blind spot.
It’s also important to realize that these problems don’t just do damage at the end of the supply chain, when it’s finally in users hands. In fact, these problems can emerge and wreak destruction at any part throughout it – especially because different links on that chain can also be characterized as users as well. Businesses need to think of themselves both as potential victims as well as potential origins of new problems.
There is a limited amount an individual business can do to combat this problem on an systemic level – it’s a function of the incredible demand for software and the lack of broader guardrails across borders, sectors and businesses. That said, making sure that they don’t become a victim or originator of software supply chain insecurity is a comparatively simple task.
It can start with a robust AppSec programme which can provide an accurate picture of the entire threat landscape with continuous automated scanning which is integrated in CI/CD workflow so it can pick up on vulnerabilities as they emerge in the software development process. On top of that, a Zero Trust approach will help enormously in mitigating supply chain risk, making sure that entities assets and third party components are examined thoroughly throughout development and treated with the correct amount of suspicion to offset risks.
The price of not dealing with these risks are well known. However, even if the threat of cyberattack, reputational damage, lost revenue or customer flight don’t prompt businesses to action, then regulation just might. The European Union’s NIS2 is on the horizon, coming into enforcement by October 2024. Much like the General Data Protection Regulation, NIS2 comes with heavy fines for the non-compliant. Unlike the GDPR, however, it makes compliant organizations account for the security of their supply chains. This should underline the need for each individual business and organization to take account for the underlying security of their software providers and partner organizations.
The software supply chain is a channel on which we all rely. As a result, each link in that chain is only as good as the links it connects to. It is incumbent upon every party within it to thoroughly assess the security of the software they both produce and receive. This is not merely a matter of personal interest for businesses, but personal integrity too.
About the Author
Frank Catucci is CTO and Head of Security Research at Invicti. He is a global application security technical leader with over 20 years of experience, designing scalable application security specific architecture, partnering with cross-functional engineering and product teams. Frank is a past OWASP Chapter President and contributor to the OWASP bug bounty initiative and most recently was the Head of Application & Product Security at Data Robot. Prior to that role, Frank was the Sr. Director of Application Security & DevSecOps and Security Researcher at Gartner, and was also the Director of Application Security for Qualys. Outside of work and hacking things, Frank and his wife maintain a family farm. He is an avid outdoors fan and loves all types of fishing, boating, watersports, hiking, camping and especially dirt bikes and motorcycles.
Frank can be reached online at [email protected] and at our company website https://www.invicti.com/