Fortinet FortiGuard Incident Response on TeamCity CI/CD CVE attack in AWS

[namle]

Overview

One of Fortinet customers, a large fintech organization, leverages TeamCity to deploy auto-scaling EC2 workload in their AWS environment. The customer builds new resources by automatically starting and stopping cloud-hosted agents on-demand, depending on the current build queue workload.

They also use FortiCNAPP (Lacework) to protect their cloud native applications and resources. During periodic scan, FortiCNAPP discovered 10% of the EC2 workload from certain AMI images expose to CVE-2023-42793 which has the score of 9.8 where attackers had already deployed the publicly available exploit without authentication supporting remote code execution on the victim server using a basic web request to any accessible web server hosting the vulnerable application.

Incident Summary

• Attack Vector: CVE exploitation on AWS workloads
• Impact: Remote code execution for unauthenticated users, enabling access to critical applications running on EC2
• Initial Entry Point: Application accessed via public IP
• Detection: CVE scanning and AWS Security Hub detection alerts

Key Findings

The findings were forwarded to AWS Security Hub through a process which utilizes Amazon EventBridge, Amazon SQS, and AWS Lambda. AWS Security Hub then triggered an incident response which logged an incident in AWS Security Incident Response (SIR).

The customer also uses FortiSOAR managed by Fortinet IR team with AWS Security Hub connector. Once AWS Security Hub triggered an IR event to FortiSOAR, the Fortinet IR team started the IR process. A notification was sent to the customer, and investigation was started to track down malicious activity in the affected AWS accounts.

Vulnerability Exploitation

During a scoping call, the FortiGuard IR team identified that one of the applications hosted on this server was TeamCity. The victim had only recently updated the application to a non-vulnerable version.

We began by retrieving application and system logs from the suspected compromised server (HOST_1_TEAMCITY). On analysis of the application logs, we identified significant evidence of successful exploitation of the TeamCity vulnerability

Analysis of these logs showed that this vulnerability had been exploited multiple times over a relatively short period, with connections originating from multiple unique public IP addresses. The teamcity-auth.log (authentication events log) identifies successful exploitation but does not provide details on commands executed through exploitation. This information is available in the separate 'teamcity-server.log' file, a general server log for the TeamCity software.

Main Threat Actor Intrusion

The first activity attributed to the main threat actor was the execution of an echo command like those discussed above, indicating that the main threat actor likely employed Nuclei to identify potential victims. After this initial command, the main threat actor began executing additional discovery commands to gather system and privilege information. Some of these discovery commands are shown below:

cmd.exe "/c systeminfo"
whoami
ipconfig /all
whoami /all

The command attributed to the Nuclei scanning, as well as these subsequent discovery commands, were linked to different remote IP addresses. However, we assessed them as being from the same actor due to the slight timeline difference of a few seconds between the activities. This indicates the main threat actor uses different infrastructures to scan for victims and execute later commands.

Response and Remediation

The Fortinet IR team coordinated a multi-phased incident response:

Given our understanding of the main threat actor’s operations, we determined that persistence was still possible. The IR team provided recommendations on removing existing adversary accesses and persistence. A high-level view of the containment and eradication actions recommended are provided below:

1. Blocking the IP addresses used by threat actors
2. Removing TeamCity software accounts created by threat actors
3. Removing Windows accounts created by threat actors
4. Removing backdoors created by threat actors
5. Removing malicious files dropped by threat actors

After implementing these containment and remediation actions by the victim security team, no further malicious activity has been observed.

Fortinet Protections:

Fortinet advised the customers to implement multi-layer approach to protect their AWS environment. The customer was able to gain visibility in the detection, analysis, containment, eradication, and recovery process. AWS SIR helped the customer and FortiGuard IR team collaborate and eliminate blind spots during the triage process.

The malware described in this report are detected and blocked by FortiGuard Antivirus as:

AntiVirus: W64/GraphicalProton.A!tr
AntiVirus: W64/Dukes.O!tr
AntiVirus: W32/Dukes.P!tr
AntiVirus: W32/PossibleThreat

FortiGate, FortiMail, FortiClient, and FortiEDR support the FortiGuard AntiVirus service. The FortiGuard AntiVirus engine is a part of each of those solutions. As a result, customers who have these products with up-to-date protections are protected.

The Fortinet Difference

Fortinet’s rapid response, powered by its integrated cloud security solutions and FortiGuard Threat Intelligence, played a decisive role in helping the customer contain and recover from the SSRF attack. The unique combination of real-time threat detection, deep cloud-native visibility, and automated response allowed the customer to:

• Contain the breach quickly and prevent further credential misuse
• Leverage FortiCNAPP and FortiGuard Threat Intelligence to identify and block malicious activity in AWS.
• Assurance of AWS environment configuration by enforcing best practices such as IMDSv2 and least privilege access

Improve cloud governance and threat detection through continuous monitoring with FortiCNAPP platform.

• Develop robust playbooks and response strategies, reducing the time to detect and respond to future threats