From CI/CD to Cloud Data: How Shai Hulud Persistence Leads to Redshift Breach

What Happened

A malicious software worm called Shai Hulud, linked to a group called TeamPCP, has been infecting software build pipelines by sneaking harmful code into popular programming packages. Once inside a build system (like Jenkins), the worm steals cloud access keys and uses them to break into the organization's cloud databases. In one confirmed case, the attackers stole credentials from a Jenkins build server, created a new administrator account in the cloud environment, opened network pathways to an Amazon Redshift data warehouse, and extracted sensitive data. Any organization using modern software build pipelines and Amazon Web Services could be affected. This matters because a single infected software dependency can lead to a full breach of production cloud databases containing sensitive customer or business data. Organizations should review their build pipeline permissions, monitor for cloud credentials being used from unexpected locations, and audit their package dependencies for known malicious versions.

Key Takeaways

• Shai Hulud supply chain worm (attributed to TeamPCP) compromises CI/CD pipelines by injecting malicious npm/PyPI packages that harvest build credentials and bridge into production cloud infrastructure.
• Attackers stole Jenkins EC2 instance role credentials via IMDS, used them from external IPs, then escalated to full cloud admin by creating IAM user 'cloudops-monitor' with AdministratorAccess.
• Attacker modified Aurora RDS and Redshift cluster security groups to open network paths to data warehouses, enumerated Secrets Manager for warehouse credentials, and exfiltrated data via Redshift Data API.
• Explicit attacker naming conventions (policies named 'exfil-s3-*', STS session names 'exfil', 'exfil10', 'exfil12') provide high-fidelity hunting and detection opportunities.
• Earliest cloud detection signal was external use of Jenkins instance credentials from non-AWS IP addresses, detectable via CloudTrail and identity correlation.

Affected Systems

• Amazon Web Services (AWS) EC2
• AWS IAM
• AWS Redshift
• AWS Aurora PostgreSQL (RDS)
• AWS Secrets Manager
• AWS S3
• AWS SES
• AWS SSM
• Jenkins CI/CD runners
• GitHub Actions (OIDC publishing abuse)
• npm and PyPI package registries

Vulnerabilities (CVEs)

None identified.

Attack Chain

1. Initial Access: Shai Hulud worm compromises CI/CD pipeline via malicious npm/PyPI package that executes during install or CI job, harvesting build credentials including AWS instance role tokens
2. Credential Access: Attacker retrieves Jenkins EC2 instance role credentials via IMDS curl on the compromised build runner
3. Privilege Escalation: Stolen instance credentials used from external IP to create IAM user 'cloudops-monitor' with AdministratorAccess policy and active access keys
4. Persistence & Discovery: Attacker creates security groups with SSH open to 0.0.0.0/0, launches staging EC2 instances, enumerates VPC subnets, RDS clusters, and Redshift warehouses
5. Collection: Attacker enumerates hundreds of Secrets Manager secrets, retrieves Redshift warehouse credentials via GetClusterCredentials, and executes ~90 Redshift Data API ExecuteStatement cycles for data collection
6. Exfiltration: Attacker attaches inline S3 policies named 'exfil-s3-*' to analytics role, assumes break-glass role with session name 'exfil', and runs SSM SendCommand on managed instances for in-VPC data transfer

Detection Availability

• YARA Rules: No
• Sigma Rules: No
• Snort/Suricata Rules: No
• KQL Queries: No
• Splunk SPL Queries: No
• EQL Queries: No
• Other Detection Logic: No
• Platforms: FortiCNAPP, FortiGuard Antivirus, FortiGuard Web Filtering, FortiGuard IP Reputation and AntiBotnet Security Service

FortiCNAPP provides cloud-native detection via CloudTrail correlation and identity context analysis, producing alerts such as 'Potentially Compromised Keys' with 1,095 observations across impossible travel, MITRE discovery, sensitive infrastructure discovery, and credential access categories. FortiGuard Antivirus detects malware components as JS/Agent.B067!tr and JS/Agent.3A86!tr. No standalone YARA, Sigma, Snort, or KQL rules are provided in the article.

Detection Engineering Assessment

Required Log Sources

• AWS CloudTrail (all regions, all services)
• AWS CloudTrail Insights
• Jenkins build logs and system logs
• EDR process execution logs on CI/CD runners
• Network egress firewall logs from build environment
• AWS VPC Flow Logs
• AWS Config change history

Hunting Hypotheses

Control Gaps

• Traditional network security controls do not detect AWS API calls made from external IPs using stolen instance credentials, as these calls use legitimate AWS endpoints over HTTPS.
• Signature-based endpoint protection on CI/CD runners may not detect credential theft via standard tools like curl accessing IMDS, as this is a legitimate administrative action.
• Cloud IAM policies allowing CI/CD instance roles to create IAM users or attach AdministratorAccess represent a privilege escalation path that least-privilege enforcement would block.
• Security group change control for database clusters (Aurora, Redshift) may not be enforced, allowing attackers to open network paths to data tiers without triggering alerts.
• Secrets Manager access by CI/CD roles with broad ListSecrets/GetSecretValue permissions enables mass credential enumeration without additional authorization checks.

Key Behavioral Indicators

• EC2 instance role credentials (assumed-role session) used from source IP addresses outside AWS CIDR ranges
• Curl process accessing IMDS endpoint on CI/CD runner followed by outbound connection to non-standard port (e.g., 8080)
• Newly created IAM user immediately receiving AdministratorAccess policy from an EC2 instance role principal
• STS AssumeRole calls with roleSessionName containing 'exfil' or similar explicit operational naming
• Inline IAM policies with names containing 'exfil-s3' attached to existing analytics or data roles
• Rapid sequence of ModifyDBCluster/ModifyCluster followed by AuthorizeSecurityGroupIngress on ports 5432 or 5439
• High-volume paginated ListSecrets calls followed by targeted GetSecretValue for warehouse-related secret names
• Redshift Data API ExecuteStatement cycles (~90 execute/describe/result loops) from external IPs
• AWS CLI user-agent strings from Qubes/Fedora distributions, which are atypical for enterprise cloud operations
• SES VerifyEmailIdentity and GetSendQuota calls from newly created IAM users or in-account EC2 instances

False Positive Assessment

Recommendations

Immediate Mitigation

Infrastructure Hardening

• Consider treating CI/CD runners as tier-0 identity by applying least-privilege instance profiles with no admin paths, no broad Secrets Manager access, and no datastore admin permissions from build roles.
• Evaluate whether IMDSv2 (token-based metadata service) is enforced on all EC2 instances, especially CI/CD runners, to prevent simple curl-based credential theft.
• Consider implementing AWS Service Control Policies (SCPs) that deny IAM user creation and AdministratorAccess policy attachment from EC2 instance roles.
• Consider enforcing change control and alerting on ModifyDBCluster, ModifyCluster, and AuthorizeSecurityGroupIngress for database ports (5432, 5439) in production environments.
• Evaluate whether your CI/CD runners have network egress restrictions that would flag or block unexpected outbound connections to non-standard ports.
• Consider implementing AWS CloudTrail Insights and automated correlation for impossible travel detection on cloud identities.

User Protection

• Consider deploying endpoint detection on all CI/CD runners that alerts on curl access to IMDS endpoints and unusual outbound network connections.
• If your organization uses Jenkins, consider reviewing whether the Jenkins instance role has been accessed from external IPs and implement alerts for this pattern.
• Consider implementing package dependency scanning in CI/CD pipelines to detect known malicious npm/PyPI package versions associated with the Shai Hulud campaign.
• Evaluate whether your CI/CD pipelines use trusted OIDC publishing in GitHub Actions and review for potential abuse of attested artifacts from attacker-influenced runners.

Security Awareness

• Consider incorporating supply chain attack scenarios into existing security awareness training, emphasizing that a single malicious package dependency can lead to full cloud data breach.
• Consider running tabletop exercises simulating the attack path from package compromise to warehouse exfiltration to test detection and response capabilities.
• If applicable, consider recommending Fortinet Certified Fundamentals (FCF) training in Cybersecurity for end users to help identify phishing attacks that may follow SES staging activity.

MITRE ATT&CK Mapping

Additional IOCs

• Command Lines:
  • Purpose: Retrieve IAM instance role credentials from EC2 Instance Metadata Service (IMDS) for theft and external reuse | Tools: curl | Stage: Credential Access | curl -s hxxp://169[.]254[.]169[.]254/latest/meta-data/iam/security-credentials/
• Other:
  • cloudops-monitor - Attacker-created IAM user with AdministratorAccess policy attached; used as primary identity for post-escalation cloud activity including Redshift data exfiltration
  • exfil - STS role session name used when assuming a break-glass/super-admin role after multiple denials; explicit attacker naming convention for exfiltration activity
  • exfil10 - STS role session name used for SSM SendCommand and follow-on activity during exfiltration phase
  • exfil12 - STS role session name used for SSM SendCommand and follow-on activity during exfiltration phase
  • exfil-s3-write - Attacker-created inline IAM policy granting PutObject, GetObject, ListBucket on a production data bucket; attached to an existing analytics role for staging exfiltration
  • exfil-s3-full - Attacker-created inline IAM policy granting full object lifecycle control (adds DeleteObject) on a production data bucket; attached to an existing analytics role
  • arn:aws:iam::240132531964:role/bus-litellm - Compromised AWS role ARN observed in FortiCNAPP alert for potentially compromised credentials; associated with attacker IP 72.224.240.111
  • JS/Agent.B067!tr - FortiGuard Antivirus detection name for malware components associated with the Shai Hulud worm campaign
  • JS/Agent.3A86!tr - FortiGuard Antivirus detection name for malware components associated with the Shai Hulud worm campaign
  • aws-cli/2.31.35 md/awscrt#1.0.0.dev0 ua/2.1 os/linux#7.0.0-15-generic md/arch#x86_64 lang/python#3.14.4 md/pyimpl#CPython ... md/distrib#ubuntu.26 - Dominant user-agent string from Ubuntu 26 AWS CLI build used by operator from IP 89.22.231.63 for most post-escalation cloud activity
  • aws-cli/2.34.29 md/awscrt#0.31.2 ua/2.1 os/linux#6.18.15-1.qubes.fc41.x86_64 md/arch#x86_64 lang/python#3.14.3 md/pyimpl#CPython ... md/distrib#fedora.42 - User-agent string from Fedora/Qubes AWS CLI build used during early Jenkins-role abuse from IP 185.204.1.225; suggests multiple operators or toolchain change