Loading...
Loading...
AI red teaming is the practice of adversarially testing AI systems — LLMs, retrieval apps, and autonomous agents — for the failures normal testing misses: prompt injection, jailbreaks, data exfiltration, and tool abuse. Practitioners map every finding to three frameworks (the OWASP Top 10 for LLM Applications, MITRE ATLAS, and the NIST AI Risk Management Framework) and report attack success rates rather than pass or fail, because model behavior is probabilistic.
In June 2025, a single email compromised Microsoft 365 Copilot. No link was clicked and no attachment was opened. The message carried hidden instructions; Copilot read them while answering an unrelated question, pulled internal files, and leaked them to an outside server. The flaw, tracked as CVE-2025-32711 and nicknamed EchoLeak, was found by a red team before an attacker used it in the wild.
That is the work. AI red teaming is not penetration testing with a new payload list. It has its own attack surface and its own frameworks, and because the target is probabilistic, its own definition of what "success" even means. Get any one of those wrong and you hand over a clean report on a system an attacker walks straight through.
AI red teaming is structured adversarial testing of an AI system — a model, the application around it, or an autonomous agent — to find behavior that violates its design: leaking data, following planted instructions, or misusing connected tools. Findings map to frameworks such as the OWASP Top 10 for LLM Applications and MITRE ATLAS, and get reported as attack success rates rather than a single pass or fail.
The scope widens fast as you climb the stack. Testing a raw model means probing jailbreak resistance, bias, and whether it will regurgitate training data. Testing an application (a support chatbot, a retrieval-augmented assistant) adds the prompt template, the knowledge base, output handling, and every integration behind it. Testing an agent adds tools, memory, and autonomy: once a model can browse, query a database, send email, or run code, a few words of planted text stop being a curiosity and become a live exploit.
Two confusions are worth clearing up. AI red teaming is broader than adversarial machine learning, which is one set of techniques it borrows — crafting inputs that fool a model. And it is not a penetration test. A pentest targets deterministic, reproducible bugs and asks "can we get in?" AI red teaming asks "can we make this system do the wrong thing?" against a target that is probabilistic: the same prompt can succeed on one run and fail on the next. That single property breaks pass/fail. A vulnerability that fires once in fifty attempts is still a vulnerability, so you stop counting bugs and start measuring how often you can trigger them.
| Dimension | Penetration testing | AI red teaming |
|---|---|---|
| Question asked | "Can we get in?" | "Can we make it do the wrong thing?" |
| Target | Deterministic system | Probabilistic model, app, and agent |
| Reproducibility | Same exploit repeats | Same input can pass or fail across runs |
| Result | Pass/fail, discrete findings | Attack success rate over repeated trials |
| Frameworks | OWASP Web, CWE, OSCP-style | OWASP LLM Top 10, MITRE ATLAS, NIST AI RMF |
Most guides name one framework and stop. The teams that move fastest use three together, because each answers a different question: what to test, how attackers do it, and how to govern the program around both.
| Framework | Owner | What it gives you | Use it to |
|---|---|---|---|
| OWASP Top 10 for LLM Applications (2025) | OWASP Gen AI Security Project | The vulnerability checklist for LLM apps (LLM01–LLM10) | Define coverage and tag every finding |
| MITRE ATLAS | MITRE | An ATT&CK-style catalog of real adversary tactics against AI | Model the attacker and label techniques |
| NIST AI RMF (AI 100-1) + AI 100-2 | NIST | Govern/Map/Measure/Manage plus an adversarial-ML taxonomy | Wrap the engagement in governance and terms |
The OWASP Top 10 for LLM Applications is the checklist. Every finding you write should map to one of its 2025 entries — the common language reviewers and auditors expect:
MITRE ATLAS, the Adversarial Threat Landscape for AI Systems, is modeled on ATT&CK and catalogs the tactics and techniques used against AI, grounded in real incidents and red-team exercises. Most of its tactics mirror ATT&CK, but two have no traditional-security equivalent: AI Model Access (AML.TA0000) and AI Attack Staging (AML.TA0001). ATLAS also carries case studies with their own IDs — like AML.CS0009, the 2016 poisoning of Microsoft's Tay chatbot — useful when you need to prove to a stakeholder that an attack class is not hypothetical.
NIST supplies the wrapper. The AI Risk Management Framework (AI 100-1) organizes the program into Govern, Map, Measure, and Manage, and the adversarial-ML taxonomy (AI 100-2, updated March 2025) gives you precise terms (evasion, poisoning, privacy attacks, prompt injection) so a report reads the same to an engineer and a regulator. That precision is no longer optional. The EU AI Act (Article 55) requires adversarial testing for general-purpose models that carry systemic risk; GPAI breaches carry fines up to €15M or 3% of global turnover under Article 101, and the Act's top band of €35M or 7% is reserved for its most serious prohibited practices.
Most LLM red teaming findings cluster in the same six places, and the public record shows every one is exploitable in production, not just on a whiteboard. Map them before you throw a single payload.
Prompt injection is the defining vulnerability, and its dangerous form is indirect. Direct injection is a user typing "ignore your instructions." Indirect injection hides instructions in content the model ingests: a web page, a PDF, a calendar invite, a RAG document, an email. EchoLeak was indirect injection. So was the 2024 Slack AI flaw disclosed by PromptArmor, where a message in a public channel made the assistant leak data from private channels. When SafeBreach researchers hid instructions in a Google Calendar invite in 2025, Gemini executed them the next time the victim asked about their schedule, including opening smart-home windows and shutters.
System prompt leakage (LLM07) exposes the hidden instructions that govern behavior, sometimes with API keys or feature flags inside. The 2023 "Sydney" leak from Bing Chat, where a Stanford student coaxed the model into printing its own confidential instructions, was the first mass-public demonstration.
Sensitive information disclosure (LLM02) pulls PII, source code, or business data out of the model or its retrieval corpus, often through a chain of innocent-looking questions rather than one obvious request.
Excessive agency (LLM06) is where an agent turns your own tools against you. Give it too many, or scope them too broadly, and every individual step clears its safety check while the chain as a whole exfiltrates data. At Black Hat 2025, Zenity's "AgentFlayer" used a single poisoned document to make ChatGPT's connectors leak API keys through an auto-rendered image URL. The agentic attack surface is the fastest-growing part of this discipline for exactly this reason.
RAG and knowledge-base poisoning (LLM04, LLM08) plants malicious content in the trusted corpus, so the exploit persists across sessions and users instead of living in one prompt. Improper output handling (LLM05) is the classic sink problem: unsanitized model output flowing into SQL, a shell, or a browser turns a chatbot into an injection vector for the rest of your stack.
The attacker is already automating this. In November 2025 Anthropic disclosed GTG-1002, a state-sponsored espionage campaign that ran roughly 80–90% of the operation through an AI agent across about 30 targets. Months later, Sysdig caught what it calls the first AI-agent-driven intrusion it has observed: a public vulnerability to full database exfiltration in four pivots, in about an hour, with no human at the keyboard. That is the speed you are now racing. Red teaming is how you walk those paths first.
An engagement has the same skeleton as a traditional red team, but the content of each phase is AI-specific. Six steps, run as a loop:
Scope and rules of engagement: Decide the target tier (model, application, or agent) and the access level — black-box (API only), gray-box (endpoint plus some config), or white-box (weights, system prompt, retrieval internals). For agents, name which tools are in play and which destructive actions (sending email, writing to a database, spending money) must be sandboxed.
Threat model: Enumerate the surface: model, prompt, training data, retrieval, tools, output sinks, and supply chain. Pick the threats that matter for the use case, such as prompt injection for a public chatbot, data leakage for an internal-IP assistant, or tool abuse for an agent.
Execute, manual then automated: Start with human creativity, then scale with tooling. Real attacks are rarely one prompt; the strongest jailbreaks run multi-turn. Microsoft's Crescendo escalates gradually by referencing the model's own prior answers, and Anthropic's many-shot jailbreaking floods a long context with fake compliant examples until the model follows the pattern.
Measure statistically: Because output is probabilistic, run each attack many times and report an attack success rate (ASR) alongside the OWASP and ATLAS categories you covered and the severity of what got through. A lucky one-off bypass and a reliable one are different risks, and a binary "vulnerable" hides which you have.
Report and remediate: Tag each finding to an OWASP LLM ID and an ATLAS technique. Then be honest about remediation: you cannot fully patch prompt injection, because it exploits how models read instructions. Controls are layered, spanning input and output filtering, least-privilege tool scoping, human approval for high-impact actions, and treating every retrieved document as hostile.
Retest: A fine-tune, a new tool, or a changed knowledge base can reopen a closed finding, so treat a fixed bug as fixed only until the next model change, then run it again.
We run adversarial AI-versus-AI testing at scale, and the pattern is the one Microsoft reported after probing 100 generative-AI products: you do not need to compute gradients to break most systems, simple attacks beat clever ones, and the work is never finished. Budget for a program that runs indefinitely, not a project that ships.
Pick the wrong tool category and you will benchmark a model when you meant to attack it. AI red teaming tools do three different jobs, and conflating them wastes engagements.
| Tool | Type | What it does |
|---|---|---|
| Microsoft PyRIT | Attack generator | Python framework to automate multi-turn attacks (Crescendo, TAP, Skeleton Key) across targets |
| NVIDIA garak | Scanner | CLI "nmap for LLMs" — broad first-pass probes for jailbreaks, injection, and leakage |
| promptfoo | Attack + eval | YAML-driven red-team plugins and strategies; maps findings to OWASP, NIST, and ATLAS |
| Giskard | Attack + eval | scan() generates adversarial suites; RAGET evaluates retrieval pipelines |
| HarmBench / JailbreakBench | Benchmark | Standardized behavior sets to compare attacks and defenses head to head |
| Meta Llama Guard / Prompt Guard | Guardrail | Classifiers you test against, not with |
Attack generators (PyRIT, garak, promptfoo, Giskard, and gradient-based suffix tools like Bishop Fox's Broken Hill) actively try to break a live target. Benchmarks (HarmBench, JailbreakBench, Meta's CyberSecEval) score a model against a fixed corpus, so a high number is a comparison, not proof of production safety. Guardrails (Llama Guard, Prompt Guard, NeMo Guardrails) are defenses; they belong on the other side of the test. None of these is a one-click pentest. PyRIT is a framework you build campaigns in, and garak is broad but shallow per probe. None of them decides what a finding means or whether it ships. You still do that part by hand.
Q: What is red teaming in AI? It is structured adversarial testing of an AI system (a model, an application, or an agent) to find behavior that violates its design, such as leaking data, following planted instructions, or misusing connected tools. Findings map to frameworks like the OWASP Top 10 for LLM Applications and MITRE ATLAS and get reported as attack success rates rather than a single pass or fail.
Q: Why is AI red teaming important? AI systems fail in ways traditional testing misses: a hidden instruction in an email or document can make an LLM leak data or trigger a connected tool, as in the 2025 EchoLeak flaw (CVE-2025-32711) in Microsoft 365 Copilot. Red teaming surfaces these behavior-level failures before an attacker exploits them, and regulations such as the EU AI Act now mandate adversarial testing for general-purpose models with systemic risk.
Q: How is AI red teaming different from penetration testing? A penetration test targets deterministic, reproducible flaws and asks "can we get in?" AI red teaming targets probabilistic behavior and asks "can we make the system do the wrong thing?" The same input can succeed or fail across runs, so AI red teaming relies on repeated trials and statistical success rates instead of a binary result.
Q: Is AI red teaming the same as adversarial machine learning? No. Adversarial machine learning is one family of techniques, crafting inputs that fool a model, that AI red teaming uses. Red teaming is broader, spanning prompt injection, social-engineering-style manipulation, RAG poisoning, agent tool abuse, and system-level testing beyond the model itself.
Q: What tools are used for AI red teaming? Practitioners group them into three types: attack generators that actively probe a live target (PyRIT, garak, promptfoo, Giskard), benchmarks that score a model against a fixed corpus (HarmBench, JailbreakBench), and guardrail models like Meta's Llama Guard that you test against rather than with.
Q: How do you red team an AI agent? Scope which tools and permissions are in play, sandbox any destructive action, then test for excessive agency: whether planted instructions can chain tool calls to exfiltrate data or act without approval. Memory persistence and multi-step "delayed" actions are specific to agents and must be tested explicitly.
Q: Which frameworks should a red team follow? Use the OWASP Top 10 for LLM Applications to define coverage, MITRE ATLAS to model attacker tactics, and the NIST AI Risk Management Framework to govern and standardize the program. Together they cover what to test and how attackers actually operate, then give you the language to report both.
The teams that get value from AI red teaming treat it as an ongoing discipline and part of a broader AI security program, not a box to tick before launch. They map the attack surface first, pick the threats that matter for the use case, run attacks enough times to trust the numbers, and re-test on every model or tool change. Frameworks keep the reporting legible and tooling keeps it fast, but nobody has automated the part where you look at a passing safety check and realize the chain still walked the data out the door. That call is still yours.
AI red teaming sits alongside the rest of your offensive program: the same discipline that pressure-tests your applications and APIs now has to pressure-test the models inside them. If you are mapping where automated, continuous testing fits against manual red-team work, the AI Pentest Buyer's Scorecard lays out an evaluation framework you can use, and you can book a demo to see continuous AI-driven testing run against your own stack.
