Your AI Gateway Was a Backdoor: Inside the LiteLLM Supply Chain Compromise
A sophisticated supply chain attack by the threat actor TeamPCP compromised the popular AI proxy package LiteLLM via a previously hijacked Trivy GitHub Action. The malicious package deployed a multi-stage payload utilizing a Python .pth file to harvest extensive cloud, Kubernetes, and AI credentials, encrypt them, and exfiltrate them to attacker-controlled infrastructure while establishing a persistent remote code execution backdoor.
Authors: Peter Girnus, Fernando Tucci, Deep Patel, Simon Dulude, Ashish Verma, John Rainier Navato
Source:
Trend Micro
- domaincheckmarx[.]zoneC2 server domain used for polling and downloading stage-2 persistent payloads.
- domainmodels[.]litellm[.]cloudExfiltration C2 domain for the LiteLLM credential harvesting payload.
- domainscan[.]aquasecurtiy[.]orgTyposquatted exfiltration domain used in the upstream Trivy GitHub Action compromise.
- filenameLiteLLM_init.pthMalicious Python .pth file dropped into site-packages to trigger execution on interpreter startup.
- filename~/.config/sysmon/sysmon.pyPersistent remote code execution backdoor script installed by the payload.
- filename~/.config/systemd/user/sysmon.serviceSystemd user service created to ensure the backdoor survives reboots.
Key Takeaways
- LiteLLM versions 1.82.7 and 1.82.8 were compromised on PyPI to deploy a 3-stage payload for credential harvesting, Kubernetes lateral movement, and persistent RCE.
- The attack was part of a broader multi-ecosystem supply chain campaign by the threat actor TeamPCP, which previously compromised Trivy and Checkmarx KICS.
- The malware utilized a malicious Python .pth file (LiteLLM_init.pth) to achieve interpreter-level persistence, executing silently on any subsequent Python startup.
- Harvested secrets, including LLM API keys, AWS credentials, and Kubernetes tokens, were encrypted via AES-256/RSA-4096 and exfiltrated to attacker-controlled infrastructure.
- The campaign demonstrated high operational maturity, including a global YouTube-based kill switch and automated bot suppression of GitHub disclosure issues.
Affected Systems
- LiteLLM (versions 1.82.7 and 1.82.8)
- Python environments (via site-packages .pth execution)
- GitHub Actions CI/CD pipelines
- Kubernetes clusters
- Docker Hub (aquasec/trivy:0.69.5 and 0.69.6)
- npm ecosystem (checkmarx-util-1.0.4)
Vulnerabilities (CVEs)
- CVE-2024-3400
Attack Chain
The attack began when TeamPCP compromised the Trivy GitHub Action, stealing a PyPI publish token. They used this token to publish malicious versions of LiteLLM (1.82.7 and 1.82.8) containing a .pth file that executes automatically on Python startup. The payload decodes a 3-layer script that harvests over 50 types of secrets (AWS, K8s, SSH, AI API keys), encrypts them via AES/RSA, and exfiltrates them to models.litellm.cloud. Finally, it attempts Kubernetes lateral movement via privileged pods and deploys a persistent backdoor via a systemd service (sysmon.service) that polls a C2 server for secondary payloads.
Detection Availability
- YARA Rules: No
- Sigma Rules: No
- Snort/Suricata Rules: Yes
- KQL Queries: No
- Splunk SPL Queries: No
- EQL Queries: No
- Other Detection Logic: Yes
- Platforms: Elastic, ZDI Threat Hunting Team
Elastic has published a detection rule for unexpected .pth file creation. Snort and Suricata rules are available by contacting the ZDI Threat Hunting team.
Detection Engineering Assessment
EDR Visibility: High — The payload executes multiple suspicious child processes (openssl, curl, kubectl) and writes persistence mechanisms to disk (.pth files, systemd services) which are highly visible to EDR telemetry. Network Visibility: Medium — Exfiltration occurs over HTTPS, hiding the payload contents, but the destination domains are newly registered and the C2 servers have a distinct AdaptixC2 JARM fingerprint. Detection Difficulty: Moderate — While the initial payload runs in memory, the subsequent actions (bulk credential access, openssl encryption, curl exfiltration, and systemd persistence) create a noisy behavioral footprint that is detectable with proper tuning.
Required Log Sources
- Process Creation (Event ID 4688 / Sysmon EID 1)
- File Creation (Sysmon EID 11)
- Network Connections (Sysmon EID 3)
Hunting Hypotheses
| Hypothesis | Telemetry | ATT&CK Stage | FP Risk |
|---|---|---|---|
| Look for unexpected creation of .pth files in Python site-packages directories, which may indicate interpreter-level persistence. | File Creation (Sysmon EID 11) | Persistence | Medium |
| Detect openssl commands used to generate random keys and encrypt files, followed immediately by curl POST requests to unknown domains. | Process Creation (Sysmon EID 1) | Exfiltration | Low |
| Monitor for Python processes spawning kubectl to enumerate secrets across all namespaces or deploying privileged DaemonSets. | Process Creation (Sysmon EID 1) | Discovery/Lateral Movement | Low |
| Identify the creation of a sysmon.service systemd user unit, especially when associated with a Python script in ~/.config/sysmon/. | File Creation (Sysmon EID 11) / Process Creation (Sysmon EID 1) | Persistence | Low |
| Hunt for processes making authenticated AWS SigV4 API calls to Secrets Manager or SSM Parameter Store outside of expected IAM roles. | Cloud Trail / API Logs | Credential Access | Medium |
Control Gaps
- Lack of egress filtering for AI infrastructure
- Unrestricted CI/CD pipeline access to secrets
- Automatic pulling of unpinned dependencies in build pipelines
Key Behavioral Indicators
- Python processes spawning openssl and curl
- Bulk reading of .env, AWS credentials, and SSH keys by a single process
- Creation of systemd user services named 'System Telemetry Service'
False Positive Assessment
- Low
Recommendations
Immediate Mitigation
- Check for LiteLLM_init.pth in any Python site-packages directory.
- Rotate ALL credentials that were present as environment variables or in config files on any system where LiteLLM 1.82.7 or 1.82.8 was installed.
- Block the identified C2 domains and IP addresses at the DNS and firewall level.
- Audit npm packages for any reference to checkmarx-util or postinstall hooks that read GITHUB_ACTIONS.
Infrastructure Hardening
- Pin package versions and verify SHA256 hashes using pip install --require-hashes.
- Enforce 'npm install --ignore-scripts' in CI/CD pipelines unless postinstall scripts are explicitly reviewed.
- Implement egress filtering and domain allowlisting for AI infrastructure.
- Pin Trivy and all security scanners in your CI/CD pipeline to stable, tested versions.
User Protection
- Deploy behavioral detection for bulk credential file reads (SSH keys, cloud configs, crypto wallets) in a single process.
- Monitor for unexpected .pth file creation in Python site-packages.
Security Awareness
- Treat LLM API keys as crown jewels and rotate them immediately if any package in the dependency tree is compromised.
- Watch for fork bombs or resource exhaustion as a potential indicator of supply chain compromise.
MITRE ATT&CK Mapping
- T1059.006 - Command and Scripting Interpreter: Python
- T1543.002 - Create or Modify System Process: Systemd Service
- T1611 - Escape to Host
- T1027 - Obfuscated Files or Information
- T1552.001 - Credentials In Files
- T1552.005 - Cloud Instance Metadata API
- T1082 - System Information Discovery
- T1610 - Deploy Container
- T1005 - Data from Local System
- T1041 - Exfiltration Over C2 Channel
- T1573.001 - Encrypted Channel: Symmetric Cryptography
- T1071.001 - Application Layer Protocol: Web Protocols
- T1078 - Valid Accounts
- T1195.002 - Compromise Software Supply Chain
Additional IOCs
- Domains:
kamikaze[.]sh- Domain associated with an Iran-targeted wiper component linked to the threat actor.
- Urls:
hxxps://checkmarx[.]zone/raw- URL polled by the persistent backdoor every 50 minutes for stage-2 payloads.
- File Paths:
/tmp/.pg_state- State tracking file used by the persistence backdoor to prevent re-downloading payloads./tmp/pglog- Temporary file where the downloaded stage-2 payload is stored before execution.tpcp.tar.gz- Archive name used for bundling encrypted harvested credentials before exfiltration.
- Command Lines:
- Purpose: Execute the base64-encoded Layer 1 orchestrator payload | Tools:
python,subprocess| Stage: Execution |subprocess.Popen([sys.executable, "-c", "import base64; exec(base64.b64decode( - Purpose: Generate a random 32-byte AES session key for data encryption | Tools:
openssl| Stage: Collection/Encryption |openssl rand -out session.key 32 - Purpose: Encrypt the harvested data using AES-256-CBC | Tools:
openssl| Stage: Collection/Encryption |openssl enc -aes-256-cbc -in collected -out payload.enc - Purpose: Exfiltrate the encrypted credential bundle via HTTPS POST | Tools:
curl| Stage: Exfiltration - Purpose: Enumerate Kubernetes secrets across all namespaces | Tools:
kubectl| Stage: Discovery/Credential Access |kubectl get secrets --all-namespaces -o json - Purpose: Enable the persistent systemd service via a privileged container escape | Tools:
chroot,systemctl| Stage: Persistence/Lateral Movement |chroot /host systemctl --user daemon-reload
- Purpose: Execute the base64-encoded Layer 1 orchestrator payload | Tools:
- Other:
aquasec/trivy:0.69.5- Malicious Docker image pushed to Docker Hub during the campaign.aquasec/trivy:0.69.6- Malicious Docker image pushed to Docker Hub during the campaign.checkmarx-util-1.0.4.tgz- Malicious npm package deployed as part of the broader supply chain campaign.