AI in Vulnerability Discovery: A Call for Human Oversight and Caution
The integration of AI in vulnerability research has led to a surge in false-positive bug reports, overwhelming vendors and bug bounty programs. Human oversight remains essential to validate AI findings and maintain the integrity of the CVE ecosystem.
Key Takeaways
- AI tools used in vulnerability discovery are generating a high volume of false positives and 'AI slop' bug reports.
- The influx of unverified AI-generated reports is overwhelming bug bounty programs, leading to shutdowns like the curl utility's program.
- Human oversight and manual validation are critical to ensure the accuracy of AI-generated vulnerability reports.
- Inexperienced individuals are using AI to spam bug bounty programs for financial gain, degrading trust in third-party research.
Affected Systems
- Bug Bounty Programs
- CVE Database
- Vulnerability Triage Workflows
Attack Chain
Inexperienced individuals or researchers use AI tools like Claude Code to automatically analyze codebases for vulnerabilities. The AI generates reports based on pattern matching, often misinterpreting benign code (e.g., integer inputs) as exploitable flaws like OS Command Injection. These unverified, false-positive reports are submitted en masse to bug bounty programs or CVE databases. This floods the system, overwhelming maintainers, wasting triage resources, and degrading the overall vulnerability management process.
Detection Availability
- YARA Rules: No
- Sigma Rules: No
- Snort/Suricata Rules: No
- KQL Queries: No
- Splunk SPL Queries: No
- EQL Queries: No
- Other Detection Logic: No
No detection rules are provided as this article discusses procedural and operational challenges with AI-generated vulnerability reports rather than a cyber attack.
Detection Engineering Assessment
EDR Visibility: None — This is a procedural issue regarding vulnerability reporting, not an endpoint attack. Network Visibility: None — No malicious network traffic is associated with this reporting issue. Detection Difficulty: N/A — This is not a detectable cyber attack, but rather an abuse of reporting mechanisms.
Required Log Sources
- Web Application Firewall (WAF)
- Ticketing System Logs
Hunting Hypotheses
| Hypothesis | Telemetry | ATT&CK Stage | FP Risk |
|---|---|---|---|
| Monitor bug bounty submission portals for high-frequency submissions from single users or IP addresses, which may indicate automated AI-driven reporting. | Web Application Firewall (WAF) or Application Logs | Reconnaissance | High |
Control Gaps
- Bug bounty submission filtering
- Automated vulnerability validation
Key Behavioral Indicators
- High volume of bug bounty submissions from a single source
- Submissions lacking functional proof-of-concept (PoC) exploits
- Reports containing common LLM phrasing or hallucinated code paths
False Positive Assessment
- High
Recommendations
Immediate Mitigation
- Implement rate limiting on bug bounty submission portals.
- Require functional Proof of Concept (PoC) exploits for all vulnerability submissions to filter out AI hallucinations.
Infrastructure Hardening
- N/A
User Protection
- N/A
Security Awareness
- Train triage teams to identify common hallmarks of AI-generated false positives.
- Establish strict guidelines for researchers regarding the responsible use of AI in vulnerability discovery.
