The AI Arms Race: How Adversaries are Weaponizing AI for Speed and Scale

What Happened

Cybercriminals and state-sponsored hackers are using artificial intelligence tools to speed up their attacks, produce more convincing phishing messages, and write malware faster. The time between an attacker breaking into a system and spreading to other systems has dropped to an average of 29 minutes, putting pressure on security teams to respond more quickly. Some advanced malware now consults AI chatbots during an attack to generate commands on the fly, and at least one campaign used an AI agent to carry out most of an espionage operation with minimal human involvement. Organizations of all types are affected, especially those that cannot keep up with patching or that are adopting AI tools without fully vetting the new software dependencies they introduce. Security teams should focus on speeding up their detection and response processes, monitoring for unusual connections to AI services from their systems, and treating intelligence as something to act on immediately rather than just read about.

Key Takeaways

• Average attacker breakout time fell to 29 minutes in 2025, with the fastest case at 27 seconds; AI-enabled adversary operations rose 89% year-on-year.
• Runtime-LLM malware such as PROMPTSTEAL (LAMEHUG) and QUIETVAULT represents a new detection surface: malware querying LLM APIs mid-execution inside victim environments.
• Anthropic reported disrupting GTG-1002, assessed as the first AI-orchestrated cyber-espionage campaign, with AI performing the majority of intrusion work against ~30 targets.
• AI-assisted malware development leaves detectable residue: anomalous in-line conversational comments and emoji status markers in malware source code.
• Fraudulent North Korean IT-worker schemes generate $250M–$600M/year and are increasingly scaled using AI for persona building and face-swapping.

Affected Systems

• Organizations with local AI command-line tools installed on endpoints
• Environments where server or service accounts can reach external LLM API endpoints
• Supply chains incorporating third-party AI models, libraries, plugins, and connectors

Vulnerabilities (CVEs)

None identified.

Attack Chain

1. Reconnaissance: State-backed actors use LLMs (Gemini, ChatGPT) to synthesize OSINT, profile targets, and automate vulnerability analysis at scale
2. Social Engineering: AI generates fluent multilingual phishing content, fabricated personas, and synthetic media; North Korean IT-worker schemes use face-swapping for identity documents
3. Malware Development: Actors use LLMs to write, debug, and obfuscate code (C++, Golang) including C2 frameworks, leaving conversational comments and emoji markers as residue
4. Execution: Runtime-LLM malware (PROMPTSTEAL, QUIETVAULT) queries language models mid-execution to dynamically generate commands or hunt for secrets using local AI tools
5. Post-compromise: AI agents (GTG-1002 via Claude Code) perform multi-stage intrusions including reconnaissance, intrusion, and ransom calculation with minimal human steering

Detection Availability

• YARA Rules: No
• Sigma Rules: No
• Snort/Suricata Rules: No
• KQL Queries: No
• Splunk SPL Queries: No
• EQL Queries: No
• Other Detection Logic: No

No detection rules are provided in the article. Images describe detection indicators conceptually (outbound LLM API connections from unauthorized processes, anomalous code comments/emoji in malware, requests involving persistence/evasion/C2 inside AI environments) but no query or rule content is included.

Detection Engineering Assessment

Required Log Sources

• EDR process telemetry (process creation, network connections, command-line arguments)
• Proxy or next-gen firewall logs (outbound HTTP/HTTPS to known LLM API domains)
• DNS logs (queries to AI model provider domains)
• Cloud audit logs (API calls to AI services from service accounts)

Hunting Hypotheses

Control Gaps

• Traditional network security controls may not flag connections to legitimate AI provider API endpoints as suspicious, since these domains are commonly allowlisted.
• Signature-based AV and EDR would not detect AI-assisted social engineering or synthetic media used in phishing.
• Patch management programs may not keep pace with the accelerating volume of vulnerability disclosures driven by AI-assisted research.
• Supply chain security programs may not vet AI-specific dependencies (models, libraries, plugins, connectors) introduced through AI adoption.

Key Behavioral Indicators

• Outbound connections to LLM API endpoints from unauthorized or unexpected processes
• Unattended/automated calls to AI model providers from server or service accounts
• Local AI command-line tools invoked by unusual parent processes
• Conversational in-line comments and emoji status markers in malware source code
• Requests involving persistence, evasion, loaders, or C2 patterns inside AI agent environments
• Anomalous multilingual phishing content with consistent tone across languages suggesting AI generation

False Positive Assessment

• Medium — Outbound connections to LLM API endpoints will generate false positives in environments where developers, power users, or legitimate AI tools are in use. Distinguishing malicious AI CLI tool invocation from benign use requires baselining and context. AI-assisted code residue (comments, emoji) may appear in legitimate open-source projects.

Recommendations

Immediate Mitigation

• Verify against your organization's incident response runbook and team escalation paths before acting. Consider reviewing proxy and firewall logs for outbound connections to known LLM API endpoints from endpoints and servers where such traffic is unexpected.
• If your EDR supports process-level network monitoring, consider creating alerts for processes making outbound HTTPS connections to AI model provider domains that are not on an approved application list.
• Consider inventorying all local AI command-line tools installed across your environment and baselining their expected usage patterns to identify anomalous invocations.

Infrastructure Hardening

• Evaluate whether task-scoped, short-lived credentials can be implemented for AI agents and service accounts that need to interact with AI APIs, limiting the blast radius of compromised credentials.
• Consider implementing network segmentation policies that restrict which servers and service accounts can reach external AI model provider endpoints.
• Where supported by your tooling, consider enabling audit logging for all AI platform API calls to create a baseline for detecting anomalous agentic activity.
• Evaluate your supply chain security program's coverage of AI-specific dependencies (models, libraries, plugins, connectors) introduced through AI adoption.

User Protection

• Consider deploying enhanced phishing detection controls that account for AI-generated multilingual content with consistent tone and grammar across languages.
• If applicable, consider implementing deepfake/synthetic media detection for identity verification processes, particularly in remote hiring contexts.
• Consider enhancing background check procedures for remote IT workers to account for AI-fabricated identities and face-swapping techniques documented in North Korean IT-worker schemes.

Security Awareness

• Consider updating existing security awareness training to include examples of AI-generated phishing content, which may be more fluent and convincing than traditional lures.
• Consider incorporating guidance on verifying the authenticity of remote colleagues and IT contractors, given the documented scaling of fraudulent IT-worker schemes.
• Where relevant to your workforce, consider educating developers on the security implications of leaving AI-assisted code residue (conversational comments, emoji markers) in production codebases.

MITRE ATT&CK Mapping

• T1592 - Gather Victim Host Information
• T1589 - Gather Victim Identity Information
• T1566 - Phishing
• T1027 - Obfuscated Files or Information
• T1059 - Command and Scripting Interpreter
• T1071 - Application Layer Protocol
• T1005 - Data from Local System
• T1078 - Valid Accounts

Additional IOCs

• Other:
  • PROMPTSTEAL - Russian APT28 malware that queries an LLM during execution to generate commands on the fly; also tracked as LAMEHUG
  • QUIETVAULT - Credential stealer that checks infected systems for local AI command-line tools to run prompts hunting for further secrets
  • OSSTUN - C2 framework developed by Chinese actor APT41 with AI assistance using Gemini for C++ and Golang development