Your AI Gateway Was a Backdoor: Inside the LiteLLM Supply Chain Compromise

Key Takeaways

• LiteLLM versions 1.82.7 and 1.82.8 were compromised on PyPI to deploy a 3-stage payload for credential harvesting, Kubernetes lateral movement, and persistent RCE.
• The attack was part of a broader multi-ecosystem supply chain campaign by the threat actor TeamPCP, which previously compromised Trivy and Checkmarx KICS.
• The malware utilized a malicious Python .pth file (LiteLLM_init.pth) to achieve interpreter-level persistence, executing silently on any subsequent Python startup.
• Harvested secrets, including LLM API keys, AWS credentials, and Kubernetes tokens, were encrypted via AES-256/RSA-4096 and exfiltrated to attacker-controlled infrastructure.
• The campaign demonstrated high operational maturity, including a global YouTube-based kill switch and automated bot suppression of GitHub disclosure issues.

Affected Systems

• LiteLLM (versions 1.82.7 and 1.82.8)
• Python environments (via site-packages .pth execution)
• GitHub Actions CI/CD pipelines
• Kubernetes clusters
• Docker Hub (aquasec/trivy:0.69.5 and 0.69.6)
• npm ecosystem (checkmarx-util-1.0.4)

Vulnerabilities (CVEs)

• CVE-2024-3400

Attack Chain

The attack began when TeamPCP compromised the Trivy GitHub Action, stealing a PyPI publish token. They used this token to publish malicious versions of LiteLLM (1.82.7 and 1.82.8) containing a .pth file that executes automatically on Python startup. The payload decodes a 3-layer script that harvests over 50 types of secrets (AWS, K8s, SSH, AI API keys), encrypts them via AES/RSA, and exfiltrates them to models.litellm.cloud. Finally, it attempts Kubernetes lateral movement via privileged pods and deploys a persistent backdoor via a systemd service (sysmon.service) that polls a C2 server for secondary payloads.

Detection Availability

• YARA Rules: No
• Sigma Rules: No
• Snort/Suricata Rules: Yes
• KQL Queries: No
• Splunk SPL Queries: No
• EQL Queries: No
• Other Detection Logic: Yes
• Platforms: Elastic, ZDI Threat Hunting Team

Elastic has published a detection rule for unexpected .pth file creation. Snort and Suricata rules are available by contacting the ZDI Threat Hunting team.

Detection Engineering Assessment

EDR Visibility: High — The payload executes multiple suspicious child processes (openssl, curl, kubectl) and writes persistence mechanisms to disk (.pth files, systemd services) which are highly visible to EDR telemetry. Network Visibility: Medium — Exfiltration occurs over HTTPS, hiding the payload contents, but the destination domains are newly registered and the C2 servers have a distinct AdaptixC2 JARM fingerprint. Detection Difficulty: Moderate — While the initial payload runs in memory, the subsequent actions (bulk credential access, openssl encryption, curl exfiltration, and systemd persistence) create a noisy behavioral footprint that is detectable with proper tuning.

Required Log Sources

• Process Creation (Event ID 4688 / Sysmon EID 1)
• File Creation (Sysmon EID 11)
• Network Connections (Sysmon EID 3)

Hunting Hypotheses

Control Gaps

• Lack of egress filtering for AI infrastructure
• Unrestricted CI/CD pipeline access to secrets
• Automatic pulling of unpinned dependencies in build pipelines

Key Behavioral Indicators

• Python processes spawning openssl and curl
• Bulk reading of .env, AWS credentials, and SSH keys by a single process
• Creation of systemd user services named 'System Telemetry Service'

False Positive Assessment

• Low

Recommendations

Immediate Mitigation

• Check for LiteLLM_init.pth in any Python site-packages directory.
• Rotate ALL credentials that were present as environment variables or in config files on any system where LiteLLM 1.82.7 or 1.82.8 was installed.
• Block the identified C2 domains and IP addresses at the DNS and firewall level.
• Audit npm packages for any reference to checkmarx-util or postinstall hooks that read GITHUB_ACTIONS.

Infrastructure Hardening

• Pin package versions and verify SHA256 hashes using pip install --require-hashes.
• Enforce 'npm install --ignore-scripts' in CI/CD pipelines unless postinstall scripts are explicitly reviewed.
• Implement egress filtering and domain allowlisting for AI infrastructure.
• Pin Trivy and all security scanners in your CI/CD pipeline to stable, tested versions.

User Protection

• Deploy behavioral detection for bulk credential file reads (SSH keys, cloud configs, crypto wallets) in a single process.
• Monitor for unexpected .pth file creation in Python site-packages.

Security Awareness

• Treat LLM API keys as crown jewels and rotate them immediately if any package in the dependency tree is compromised.
• Watch for fork bombs or resource exhaustion as a potential indicator of supply chain compromise.

MITRE ATT&CK Mapping

• T1059.006 - Command and Scripting Interpreter: Python
• T1543.002 - Create or Modify System Process: Systemd Service
• T1611 - Escape to Host
• T1027 - Obfuscated Files or Information
• T1552.001 - Credentials In Files
• T1552.005 - Cloud Instance Metadata API
• T1082 - System Information Discovery
• T1610 - Deploy Container
• T1005 - Data from Local System
• T1041 - Exfiltration Over C2 Channel
• T1573.001 - Encrypted Channel: Symmetric Cryptography
• T1071.001 - Application Layer Protocol: Web Protocols
• T1078 - Valid Accounts
• T1195.002 - Compromise Software Supply Chain

Additional IOCs

• Domains:
  • kamikaze[.]sh - Domain associated with an Iran-targeted wiper component linked to the threat actor.
• Urls:
  • hxxps://checkmarx[.]zone/raw - URL polled by the persistent backdoor every 50 minutes for stage-2 payloads.
• File Paths:
  • /tmp/.pg_state - State tracking file used by the persistence backdoor to prevent re-downloading payloads.
  • /tmp/pglog - Temporary file where the downloaded stage-2 payload is stored before execution.
  • tpcp.tar.gz - Archive name used for bundling encrypted harvested credentials before exfiltration.
• Command Lines:
  • Purpose: Execute the base64-encoded Layer 1 orchestrator payload | Tools: python, subprocess | Stage: Execution | subprocess.Popen([sys.executable, "-c", "import base64; exec(base64.b64decode(
  • Purpose: Generate a random 32-byte AES session key for data encryption | Tools: openssl | Stage: Collection/Encryption | openssl rand -out session.key 32
  • Purpose: Encrypt the harvested data using AES-256-CBC | Tools: openssl | Stage: Collection/Encryption | openssl enc -aes-256-cbc -in collected -out payload.enc
  • Purpose: Exfiltrate the encrypted credential bundle via HTTPS POST | Tools: curl | Stage: Exfiltration
  • Purpose: Enumerate Kubernetes secrets across all namespaces | Tools: kubectl | Stage: Discovery/Credential Access | kubectl get secrets --all-namespaces -o json
  • Purpose: Enable the persistent systemd service via a privileged container escape | Tools: chroot, systemctl | Stage: Persistence/Lateral Movement | chroot /host systemctl --user daemon-reload
• Other:
  • aquasec/trivy:0.69.5 - Malicious Docker image pushed to Docker Hub during the campaign.
  • aquasec/trivy:0.69.6 - Malicious Docker image pushed to Docker Hub during the campaign.
  • checkmarx-util-1.0.4.tgz - Malicious npm package deployed as part of the broader supply chain campaign.