Axon Rapid Response - CVE-2023-3519 Critical RCE in Citrix Products (NetScaler ADC and NetScaler Gateway)
Incident Report for Rapid Response Status Page
Resolved
Dear customers,

We are pleased to inform you that our team has effectively assessed the risk associated with the vulnerability based on the currently available information.

If you have any questions, please do not hesitate to reach out to us.

Yours,
Team Axon
Posted Jul 26, 2023 - 06:39 UTC
Update
Recently, new technical insights on the exploitability of vulnerability CVE-2023-3519 have been released publicly. While there are still some gaps in the technical explanations and proof-of-concept, the primary susceptible API endpoint has been identified. Based on public research group findings, a function behind the weak endpoint, /gwtest/formssso, does not validate the length of a given parameter, resulting in a potential overflow and consequent RCE.

With that information, we could develop the following queries to assist you in detecting related activities.

* Citrix Netscaler logs: The query looks for the unique HTTPREQUEST event type which tracks for HTTP packets sent to Citrix appliances, extracts the relevant attributes, and searches for an extended length URI to the vulnerable endpoint /gwtest/formssso.



The query can be found in Axon's RR git:
https://github.com/axon-git/rapid-response/blob/main/Citrix%20RCE%20CVE-2023-3519/citrix_netscaler_logs_overflow_requests_to_formssso.sql

* CDN Logs: The queries track packets of extended length to the vulnerable endpoint /gwtest/formssso in the CDN or WAF logs. The products supported include CloudFlare and AWS WAF. Kindly note that this query is only applicable if your Citrix applicants are protected by the WAF.

The queries can be found in Axon's RR git:
https://github.com/axon-git/rapid-response/blob/main/Citrix%20RCE%20CVE-2023-3519/cdn_logs_overflow_requests_to_formssso.sql


Considering the technical specifics now available, our team is confident that a functional POC will be published in the near future. We strongly advise immediate patching of any impacted servers without hesitation.

Axon is always on hand to provide any technical guidance you may require. Please don't hesitate to get in touch with us if you have any further queries.

Yours,
Team Axon
Posted Jul 25, 2023 - 10:15 UTC
Update
The team is constantly tracking updates related to the recently discovered vulnerability. Despite technical specifics and a proof-of-concept remaining unavailable at this stage, community discussions have surfaced, signaling possible compromised appliances as of July 7th, roughly two weeks prior to the official patch issued by Citrix.

Based on gathered evidence, it appears the subsequent following web shells names were generated within Netscaler’s directories:

* info.php
* prod.php
* log.php
* logout.php
* vpn.php
* config.php

At present, only a single hash associated with these web-shells has been published:
logout.php 293fe23849cffb460e8d28691c640a5292fd4649b0f94a019b45cc586be83fd9.
This hash has been scanned by the team across all Axon clients for the past 60 days, and any environment identified as impacted will receive a direct notification.

Moreover, advice from Incident Response teams investigating incidents linked to CVE-2023-3519 has emphasized the following:
* Threat actors may attempt to hijack ADC’s ns.conf to decrypt Netscaler secrets. 
We thus advise rotating all Netscaler secrets, private keys, and certificates within the Netscaler appliances.

* System patching may not automatically remove web shells from compromised appliances. Therefore, inspecting the subsequent directories post-patch is recommended, ensuring no irregular files are present. Use the timestamps on these files as the starting point for your investigation.
* /var/vpn/
* /var/netscaler/logon/
* /var/python/
* /netscaler/ns_gui/


Our team is always ready to assist with pressing concerns. If you detect a questionable indicator on your Netscaler appliance, please promptly open a Jira ticket for our team to provide support.

Yours,
Team Axon.
Posted Jul 23, 2023 - 06:39 UTC
Monitoring
While specific technical information or proof of concept hasn't been made public yet, Axon is intensifying efforts to identify potential threats by investigating any suspicious actions revealed through Citrix Netscaler audit logs. This is based solely on the incomplete data currently available, and a logical assumption the team has on how the vulnerability may work.

To achieve this, we have written a query that identifies unusual commands executed under the NetScaler Service by utilizing CMD_EXECUTED events from Netscaler audit logs. The query is designed to learn from and exclude commands previously executed by the service, thereby enhancing the accuracy of its results.

You can find the query in Axon’s Rapid Response repo:
https://github.com/axon-git/rapid-response/blob/main/Citrix%20RCE%20CVE-2023-3519/suspicious_netscaler_citrix_commands.sql

Our team is actively searching for any indications of compromise of customers with the relevant Netscaler data source. In cases where we suspect customers may have been affected by the vulnerability, we will reach out to them privately.


Yours,
Team Axon.
Posted Jul 19, 2023 - 15:16 UTC
Investigating
Team Axon acknowledges the recent identification of vulnerabilities in Citrix products. On July 18th, Citrix announced the discovery of these vulnerabilities in its NetScaler Products. One Critical vulnerability which stands out is a severe RCE vulnerability that allows remote code execution without any need for authentication or user involvement, specifically on Citrix Gateway and ADC products. This vulnerability has been assigned the CVE ID - CVE-2023-3519.

Based on information from the Citrix team and intelligence firms, these vulnerabilities are actively being exploited by threat actors in the wild and are deemed critical due to their easy exploitability and the frequent public exposure of these appliances.

While our team is delving into logging and forensics evidence that may provide contextual insights for posturing and threat-hunting visibility, we strongly recommend the prompt application of the new update provided by Citrix to the relevant servers.


The vulnerabilities impact the following versions of Netscaler: 
* NetScaler ADC and NetScaler Gateway 13.1 prior to 13.1-49.13 
* NetScaler ADC and NetScaler Gateway 13.0 prior to 13.0-91.13 
* NetScaler ADC 13.1-FIPS before 13.1-37.159
* NetScaler ADC 12.1-FIPS before 12.1-55.297
* NetScaler ADC 12.1-NDcPP before 12.1-55.297
* Note: NetScaler ADC and NetScaler Gateway version 12.1 is now End Of Life (EOL) and remains vulnerable.


To mitigate the vulnerabilities, we recommend patching to the following versions:
* NetScaler ADC and NetScaler Gateway 13.1-49.13  and subsequent releases
* NetScaler ADC and NetScaler Gateway 13.0-91.13  and subsequent 13.0 releases 
* NetScaler ADC 13.1-FIPS 13.1-37.159 and subsequent 13.1-FIPS releases 
* NetScaler ADC 12.1-FIPS 12.1-55.297 and subsequent 12.1-FIPS releases 
* NetScaler ADC 12.1-NDcPP 12.1-55.297 and subsequent 12.1-NDcPP releases
* Note: Given that the NetScaler ADC and NetScaler Gateway version 12.1 is now End Of Life (EOL), customers are advised to upgrade their appliances to one of the supported versions addressing these vulnerabilities.


More details are available on the Citrix notification blog: https://support.citrix.com/article/CTX561482/citrix-adc-and-citrix-gateway-security-bulletin-for-cve20233519-cve20233466-cve20233467

Additional updates will be posted on Axon’s status page as soon as possible.


Yours,
Team Axon
Posted Jul 19, 2023 - 09:50 UTC
This incident affected: Rapid Response.