Envoy has carved out a critical role in cloud-native computing, becoming increasingly prevalent as the default ingress controller for Kubernetes. This high-performance proxy, developed by Lyft and now part of the Cloud Native Computing Foundation’s arsenal, is integral for companies scaling up their Kubernetes deployments. Envoy ensures efficient load balancing, security, and operational agility by managing external access to services within Kubernetes clusters,.
Big names in tech, such as Google, IBM, and Pivotal, rely on Envoy for its robust feature set in their Kubernetes environments. Its ability to simplify complex service-to-service communication is invaluable in dynamic service environments and highlights its capabilities as an edge and middle proxy that optimizes API management across various platforms. Employing Envoy as an ingress controller in Kubernetes offers substantial benefits by centralizing the entry points into the environment and providing fine-grained control over traffic management and security.
However, the spotlight on Envoy also brings to light significant vulnerabilities, as evidenced by the recent discovery of CVE-2024-45806, CVE-2024-45809, and CVE-2024-45810. These vulnerabilities pose risks such as unauthorized access, bypassing security controls, and potential denial of service—issues that can severely impact any system’s integrity and availability.
The versions affected by these vulnerabilities are the following:
- 1.29 (CVE-2024-45809)
- 1.32.0 (CVE-2024-45810, CVE-2024-45806)
In this article, we will explore the latest Envoy API exploits in-depth, their implications, potential risks to Kubernetes deployments, and the necessary steps to mitigate these vulnerabilities effectively.
Description
CVE-2024-45806 allows external actors to control Envoy headers, resulting in unauthorized access and other potentially malicious actions within the network. This issue emerges as a result of the default configuration of internal trust boundaries in Envoy.
By default, Envoy automatically trusted all private IP addresses (RFC1918) even if no specific configuration was set. This allowed untrusted sources on the private network to be able to inject arbitrary headers such as the “X-Envoy” header.
The vulnerability could be exploited by crafting a specific HTTP request with an X-Envoy-Original-Path header, as shown below. This could have allowed an attacker to manipulate the original path and access unauthorized resources within the network.
Remediation
To mitigate the vulnerability, a runtimeFeatureEnabled check has been added in the DefaultInternalAddressConfig class to verify whether the
envoy.reloadable_features.explicit_internal_address_config flag is enabled. If the flag is enabled, external IP addresses will return a false warning, indicating a potential unauthorized access.
Disclaimer
Starting from the next version, no IP addresses will be considered trusted by default. If you have a service that needs to be trusted, you must add the necessary addresses to the internal_address_config setting as shown below. This will ensure that only the addresses you trust are treated as internal.
Description
The second vulnerability, CVE-2024-45809, can cause the application to crash when the route cache is cleared using remote JSON Web Keys. When certain conditions are met, a crash occurs due to nullptr reference conversion from the route function.
JSON Web Keys (JWKs) represent cryptographic keys in JSON format. They are mostly used with JSON Web Tokens (JWTs) to sign and verify tokens.
Conditions of the Crash
In this particular case, application crash happened under the following conditions:
- The system used JSON Web Keys, which require asynchronous processing of headers.
- The setting to clear the cache is enabled.
- The JWT filter has features that allow modifying of headers (e.g., converting them into claims)
- The routing table is set up so that the changes made by the JWT header operations result in requests that don’t match any route.
Cause of the Crash
A sequence of operations triggers the crash:
- The JWT filter pauses and starts decoding headers.
- The router filter finds a route and begins processing the upstream request.
- The JWT filter then clears the route cache.
- When the upstream request checks for its route after clearing the cache, the expected route no longer exists, leading to a crash.
Once all these conditions are met, a crash occurs because of a null pointer reference in the code related to routing.
Remediation
The Envoy team has enhanced the authentication implementation by ensuring that the cache is appropriately cleared when JWT authentication is not verified, thereby preventing potential crashes in the Envoy Proxy.
The last vulnerability, CVE-2024-45810, caused Envoy to crash under certain circumstances due to handling of the sendLocalReply function in the HTTP async client.
The HTTP async client might crash during handling the sendLocalReply() function. The reasons for the crash are as follows:
- The HTTP async client copies/duplicates the status code.
- While this is happening, the router gets destroyed because of the async stream
- The stream is supposed to be deleted later, but it’s already being deleted first.
Even though the stream decoder is destroyed, its reference is still being accessed, which results in a segmentation fault. This happens when a program tries to access memory that it shouldn’t, causing it to crash.
This issue affects the external authorization (ex_authz) feature if these conditions are met:
- Upgrade and Connection headers are allowed
- Request mirroring should be enabled
This vulnerability could be exploited by configuring allowed_headers to match any header or, more specifically, the ‘upgrade’ and ‘connection’ headers as shown below:
patterns:
– exact: upgrade
– exact: connection
Sending websocket upgrade requests ultimately resulted in the authentication sending a 400 (Bad Request) HTTP Response to reject the auth request causing Envoy to crash.
Remediation
To fix this issue, the Envoy team has added multiple checks;
Firstly, the implementation verifies whether the response headers have already been encoded. If they have, it calls the resetStream() method and returns immediately, preventing the function from attempting to encode the headers again, which could lead to a crash.
Secondly, a new routerDestroy() function has been added to clean up the router properly.
As shown below, it is invoked in the cleanup method to guarantee that the router is properly destroyed before the stream is deleted. This prevents potential crashes or unexpected behavior during the stream’s destruction process, ensuring a safe and orderly cleanup, as illustrated in the screenshot below.
As usual, keeping your software updated is the best option for mitigating known vulnerabilities. Therefore, we strongly recommend updating Envoy to the latest version to safely remediate these vulnerabilities.
To learn more about how Wallarm can protect your APIs, request a demo with our experts.