Application programming interfaces (APIs) are essential to how generative AI (GenAI) functions with agents (e.g., calling upon them for data). But the combination of API and LLM issues coupled with rapid rollouts is likely to see numerous organizations having to combat security failings.
While GenAI is susceptible to the usual security issues associated with APIs such as authentication, authorization and data exposure, there are also AI-specific concerns which have been well-documented by the OWASP Project in its Top 10 threats to LLM Applications and Generative AI.
How AI and API attacks align
At the top of the OWASP list is prompt injection (LLM01), which sees commands in the form of malicious code implanted into the LLM to manipulate the output for purposes such as data exfiltration, etc. This attack can be performed directly via the system prompt or indirectly via external sources that the LLM consults, such as websites.
Prompt injection can be used to trick the LLM into deleting other user accounts or enumerate through users because it treats the user instruction as trusted when calling on backend APIs for the requested information. It can therefore result in the second issue on the list – insecure output handling (LLM02) – i.e., the outputs generated by the LLM that then get passed downstream. For these reasons, controls need to be put in place such as input validators, output guards and content filtering.
Another key issue relevant to APIs is model denial of service (LLM04). Similar to problems raised in the OWASP Top 10 API Security Risks – unrestricted resource consumption (API4) and unrestricted access to sensitive business flows (API6) – this sees the LLM flooded with requests. The attack effectively forces the business to dedicate resources to answering requests and therefore spend and can see a decline in the quality of service of other users. These attacks can also exploit the limitations of APIs by, for example, altering rate limiting which can then see the API overwhelmed with requests in a volumetric attack. There are number of ways for countering this threat, from capping or imposing limits on how quickly requests with multiple components are fulfilled to enforcing API rate limits, to monitoring resource utilization on the network.
Of course, one of the primary concerns with GenAI is its potential to leak sensitive information which is why we’ve seen many proprietary providers emphasize the need to lock down access privileges before deploying the technology in a commercial setting. Sensitive information disclosure (LLM06) refers to the need to limit the data the LLM has access to. This is because even restricting system prompts offers no guarantee that the LLM won’t be tricked into revealing sensitive data due to prompt injection. So, the simplest solution is to prevent data entering the training model.
What we don’t yet know is what happens if the API is compromised that is providing the data to answer that prompt? Broken object property level authorization (API3) can see APIs return sensitive information following an unauthorized request.
The final but by no means least issue in the top ten is model theft (LLM10), which refers to the compromise of an entire proprietary model and the sensitive intellectual property it is privy to. This can happen in a variety of ways, from exploiting a misconfiguration in network or system settings to a side-channel attack or via a malicious insider. But one of the examples given is also via API: The attacker could query the model API using crafted inputs and prompt injection and use the output to create a shadow model, the OWASP Project suggests. To prevent this, it’s necessary to implement strong access controls, restrict LLM access to network resources and APIs, and to rate limit APIs and add filters to reduce the risk of exfiltration.
CVEs and PoCs
Such attacks are no longer purely theoretical; a number of issues have already been documented, particularly in relation to LLM06 (sensitive information disclosure).
Back in March it was revealed that the Ray AI framework could be exploited via a job submission API due to the fact the AI does not enforce authentication or support authorization models. This is due to the way Ray had been designed with users responsible for security on the systems and services it is accessing although the intention is to add authentication later. But in the meantime, the CVE enabled calls to the API to retrieve all notes held in the cluster as well as credentials.
During the same month we also learned that Google’s Gemini LLM (formerly known as Bard), which comes in three sizes (Nano, Pro and Ultra) could also be compromised. A researcher disclosed multiple vulnerabilities and warned developers using the Gemini Pro API that “system prompts can be leaked, revealing the inner workings of the program using the LLM, potentially enabling more targeted attacks”. The findings meant businesses looking to use Gemini Pro were at risk of the LLM leaking system prompts.
Data leakage was also highlighted as an issue with the Flowise low-code LLM builder by researchers last month. They discovered an earlier version of the platform that was susceptible to authentication bypass via the REST-API endpoints on the server. The error message returned by the API effectively gave the researchers access to the API without the need for authentication and they were then able to view 959 Flowise servers, 438 of which were vulnerable.
They were then able to interact with the API endpoints and to identify data that could have been retrieved without the need for any access permissions. This included API keys, GitHub access tokens and URLs with database passwords, as well as all the configurations and LLM prompts associated with the applications.
Defending against attacks
To help address these issues, LLM providers are red team testing their systems to detect and train the LLM on how to defend against the attack types identified in the OWASP Top 10.
But while testing LLMs is critical to see if vulnerabilities can be exploited and/or data exfiltrated, it’s not sufficient. To complete the lifecycle, one should monitor all activity to the APIs servicing LLMs to ensure sensitive data exposure is limited to authenticated and authorized viewers. Runtime monitoring should extend to vulnerability exploits and the even more dangerous threat of business logic abuse, where attackers look to exfiltrate data through exploring the business logic and flaws within it – LLMs will expose new flaws. Given the data that can be in use in such scenarios, rapid inline native response is critical for any security platform.
But it’s not just attacks that seek to gain access to data that they need to worry about; the infrastructure itself is also a prime target as described under LLM10 (model theft).
The NCSC recently warned that the only thing holding back organized criminal gangs in harnessing AI is access to training models, and security researchers recently found that they are getting around this by resorting to LLMjacking. They discovered that many cloud-hosted LLMs, which allow developers to rapidly spin up an LLM, were being targeted by attackers who sought to determine access credentials and quotas, presumably to offer access to them for a price. The LLM could then be utilized at the expense of the client, and the researchers claim this could cost up to $46,000 a day. How were the attackers able to access the LLMs? By using legitimate API requests and testing the limits or parameters that could be tolerated without being detected.
LLMs are clearly an attractive target due to their ability to access sensitive data and their processing capabilities, which makes access to the model lucrative in its own right. Relying on instructing third-party tools to respond will be too late and the malicious actor will already have moved, probably using the LLMs themselves. Whilst LLMs allow business to enhance functionality and customer experience, they also open new opportunities for attackers to exploit. There is no short cut here and those using LLMs should not assume all is secure.