Layer 4 of 8, Secure what you build and buy

Testing and red teaming: attack your own AI on purpose, before and after release

This layer attacks and evaluates your own models, apps and agents on purpose, with scanners, generated and multi-turn attacks, evaluations and expert red teams, before release and after each change.

24 tools + 9 also cover it
Last reviewed October 21, 2026

See the tools How the eight layers work

Executive summary

What it is: This layer attacks and evaluates your own models, apps and agents on purpose, with scanners, generated and multi-turn attacks, evaluations and expert red teams, before release and after each change.
The risk: Prompt injection, data leakage, system prompt extraction and agent tool abuse that nobody looked for until an attacker did, plus third-party models whose behavior on your use case nobody checked.
What to buy or do: Run an open-source or commercial scanner in CI against every LLM app, block releases on regressions, and buy an expert or managed red team for high-risk agents. Map findings to OWASP ids so each fix reaches the right layer.

Our analysis Our editorial model, not a standard: OWASP, MITRE and NIST did not create or endorse it. Layer placements are our analysis, based on documented capabilities. How the model works.

The question this layer answers

Have we attacked this model, app or agent the way an adversary would, and do we repeat it after each change?

Budget line
AI red teaming tools or a managed red team service, usually in offensive security or application security testing; evaluation frameworks often sit with ML teams.

What it protects against

Our analysis

Prompt injection and jailbreak weaknesses
Data leakage and system prompt extraction
Harmful or false output
Agent tool abuse
Regressions after model, prompt or tool changes
Unverified behavior of third-party models
Bias and fairness problems

Typical controls

Our analysis

Automated vulnerability scans with probe libraries
Generated and multi-turn attacks
Agent and tool abuse tests
Evaluation harnesses and benchmarks
CI regression gates
Reports mapped to frameworks
Expert-led or managed red teaming
Adversarial ML attacks on classic models
Fairness and bias evaluations

How it maps to standards

Source states means the framework's own text names the control or outcome this layer delivers. Our analysis means we drew the link. OWASP LLM ids are the 2025 edition; the ATLAS hub holds the MITRE data version.

Testing and red teaming: standards entries, basis, where the source says it and how it applies
Framework	Entry	Basis	Where the source says it	How it applies
OWASP LLM Top 10 (2025)	LLM01 Prompt Injection	Source states	Prevention list	Run regular adversarial tests and treat the model as an untrusted user.
OWASP LLM Top 10 (2025)	LLM03 Supply Chain	Source states	Prevention list	Red teaming and evaluations on third-party models for your own use cases.
OWASP LLM Top 10 (2025)	LLM04 Data and Model Poisoning	Source states	Prevention list	Red-team the model against adversarial techniques.
OWASP LLM Top 10 (2025)	LLM02 Sensitive Information Disclosure	Our analysis	Not named by the source	Tools document probes for data leakage. OWASP's prevention list for this risk does not name testing.
OWASP LLM Top 10 (2025)	LLM05 Improper Output Handling	Our analysis	Not named by the source	Testing can surface unsafe output reaching other systems. OWASP's prevention list for this risk does not name testing.
OWASP LLM Top 10 (2025)	LLM06 Excessive Agency	Our analysis	Not named by the source	Tools document tests for agent tool abuse. OWASP names SAST and dynamic application testing in development pipelines (item 8), not adversarial testing of agent tool use.
OWASP LLM Top 10 (2025)	LLM07 System Prompt Leakage	Our analysis	Not named by the source	Tools document probes for system prompt extraction. OWASP's prevention list for this risk does not name testing.
OWASP LLM Top 10 (2025)	LLM09 Misinformation	Our analysis	Not named by the source	Tools document tests for false or misleading output. OWASP's prevention list for this risk does not name testing.
OWASP Agentic Top 10 (2026)	ASI01 Agent Goal Hijack (opens in a new tab) OWASP relates it to LLM01, LLM06.	Source states	Mitigation 8	Periodic red-team tests simulating goal override.
OWASP Agentic Top 10 (2026)	ASI05 Unexpected Code Execution (RCE) (opens in a new tab) OWASP relates it to LLM01, LLM05.	Source states	Mitigation 2	Security evaluations and adversarial unit tests before production.
OWASP Agentic Top 10 (2026)	ASI06 Memory & Context Poisoning (opens in a new tab) OWASP relates it to LLM01, LLM04, LLM08.	Source states	Mitigation 7	Adversarial tests, snapshots and rollback.
OWASP Agentic Top 10 (2026)	ASI08 Cascading Failures (opens in a new tab) OWASP relates it to LLM01, LLM04, LLM06.	Source states	Mitigation 9	Replay recorded agent actions in an isolated clone to check for cascades.
OWASP Agentic Top 10 (2026)	ASI02 Tool Misuse and Exploitation (opens in a new tab) OWASP relates it to LLM06.	Our analysis	Not named by the source	Agent tool abuse tests.
MITRE ATLAS	AML.T0051 LLM Prompt Injection	Our analysis	Not named by the source	Red team tools emulate this technique.
MITRE ATLAS	AML.T0054 LLM Jailbreak	Our analysis	Not named by the source	Red team tools emulate this technique.
MITRE ATLAS	AML.T0056 Extract LLM System Prompt	Our analysis	Not named by the source	Red team tools emulate this technique.
MITRE ATLAS	AML.T0057 LLM Data Leakage	Our analysis	Not named by the source	Red team tools emulate this technique.
MITRE ATLAS	AML.T0053 AI Agent Tool Invocation	Our analysis	Not named by the source	Red team tools emulate this technique.
MITRE ATLAS	AML.T0024 Exfiltration via AI Inference API	Our analysis	Not named by the source	Red team tools emulate this technique.
NIST AI RMF	MEASURE Measure (opens in a new tab)	Source states	MEASURE 1.3, 2.7, 2.11	MEASURE uses testing and assessment methods, with independent assessors (1.3), security and resilience evaluation (2.7) and fairness and bias evaluation (2.11).
NIST AI RMF	GOVERN Govern (opens in a new tab)	Source states	GOVERN 4.3	Organizational practices enable AI testing.

Tools in this layer (24)

Tools whose category is AI red teaming & LLM evaluation, listed by name. Compare them with Coverage Scores on the Red teaming hub.

Adversa AI

Adversa AI · AI red teaming and agent security platform

Commercial
Adversarial Robustness Toolbox

LF AI & Data Foundation · open-source adversarial machine learning library
6,283 GitHub stars as of Oct 21, 2026; last push Oct 8, 2026

Open sourceMIT
Agentic Radar

Zscaler (SPLX) · open-source scanner for agent workflows
1,058 GitHub stars as of Oct 21, 2026; last push Nov 27, 2025

Open sourceApache-2.0
Agentic Security

msoedov (open-source maintainer) · open-source LLM vulnerability scanner
2,024 GitHub stars as of Oct 21, 2026; last push Oct 19, 2026

Open sourceApache-2.0
AI Red Teaming Agent (Microsoft Foundry)

Microsoft · automated red teaming tool in Microsoft Foundry

Commercial
Augustus

Praetorian · open-source Go LLM vulnerability scanner
309 GitHub stars as of Oct 21, 2026; last push Oct 20, 2026

Open sourceApache-2.0
b3: Backbone Breaker Benchmark

Check Point (Lakera) and UK AI Security Institute · agent backbone LLM security benchmark

Open sourceMIT
DeepTeam

Confident AI · open-source LLM red teaming framework
3,035 GitHub stars as of Oct 21, 2026; last push Oct 20, 2026

Open sourceApache-2.0
Enkrypt AI

Anaconda (Enkrypt AI) · AI red teaming platform with guardrails

Commercial Freemium
FuzzyAI

Palo Alto Networks (CyberArk) · open-source LLM jailbreak fuzzer
1,587 GitHub stars as of Oct 21, 2026; last push Feb 6, 2026

Open sourceApache-2.0
garak

NVIDIA · open-source LLM vulnerability scanner
9,636 GitHub stars as of Oct 21, 2026; last push Oct 21, 2026

Open sourceApache-2.0
Giskard

Giskard AI · open-source LLM agent testing library
5,916 GitHub stars as of Oct 21, 2026; last push Oct 20, 2026

Open sourceApache-2.0 Freemium
Gray Swan

Gray Swan · AI red teaming and agent assessment platform

Commercial
HarmBench

Center for AI Safety · open-source red teaming benchmark
1,076 GitHub stars as of Oct 21, 2026; last push Aug 16, 2024 inactive

Open sourceMIT
Inspect

UK AI Security Institute · open-source LLM evaluation framework
3,058 GitHub stars as of Oct 21, 2026; last push Oct 21, 2026

Open sourceMIT
Lakera Red

Check Point (Lakera) · automated and expert AI red teaming offering

Commercial
Mindgard

Mindgard · AI red teaming platform

Commercial
Promptfoo

OpenAI (Promptfoo) · open-source LLM eval and red teaming tool
26,250 GitHub stars as of Oct 21, 2026; last push Oct 22, 2026

Open sourceMIT Freemium
promptmap

utkusen · open-source prompt injection scanner
1,288 GitHub stars as of Oct 21, 2026; last push Dec 1, 2025

Open sourceGPL-3.0
PyRIT

Microsoft · open-source generative AI red teaming framework
4,668 GitHub stars as of Oct 21, 2026; last push Oct 22, 2026

Open sourceMIT
Repello AI

Repello AI · autonomous AI red teaming platform

Commercial
Spikee

Reversec · open-source prompt injection testing toolkit
268 GitHub stars as of Oct 21, 2026; last push Sep 11, 2026

Open sourceApache-2.0
SPLX

Zscaler (SPLX) · AI red teaming platform

Commercial
TextAttack

QData · open-source NLP adversarial attack framework
3,475 GitHub stars as of Oct 21, 2026; last push Aug 15, 2026

Open sourceMIT

Also cover this layer (9)

Our analysis Tools from other categories that document a capability in this layer. Each row names the layer the tool belongs to first and the documented capability our placement rests on.

Tools from other categories that also cover Testing and red teaming, with the capability each placement rests on
Tool	Its primary layer	How it applies	Based on
AI Verify103 GitHub stars as of Oct 21, 2026; last push Mar 23, 2026	L1 Governance	Documents fairness and explainability tests for AI models, which is testing work.	Runs technical tests for fairness, explainability and robustness from a portal or the command line
Akto1,532 GitHub stars as of Oct 21, 2026; last push Oct 21, 2026	L7 Agent access	Documents red teaming of AI agents and MCP servers with security probes for injection, tool abuse and privilege escalation.	Red teams AI agents and MCP servers with security probes aimed at prompt injection, tool abuse and privilege escalation
Holistic AI	L1 Governance	Documents automated bias and privacy tests for AI systems.	Automated tests for bias, robustness, privacy and transparency
IBM watsonx.governance	L1 Governance	Documents quality and fairness evaluations of AI models.	Evaluates machine learning models for quality, fairness and accuracy
ModelOp Center	L1 Governance	Documents automated bias, drift and performance tests on models.	Automated tests for bias, drift and performance
NeuralTrust15 GitHub stars as of Oct 21, 2026; last push Oct 21, 2026	L7 Agent access	Documents TrustTest, which covers prompt injection, jailbreaks, data extraction and multi-turn agentic attacks.	TrustTest runs adversarial tests for prompt injection, jailbreaks, data extraction and multi-turn agentic attacks
Operant AI	L7 Agent access	Documents a red teaming command line tool for agents.	Publishes Woodpecker, a modular red teaming CLI for AI and cloud apps
Pillar Security	L7 Agent access	Documents multi-turn adversarial attacks on agents.	Runs multi-turn adversarial attacks with attack transcripts and reproduction steps
Straiker	L7 Agent access	Documents continuous red teaming of agents across tools, MCP servers and workflows.	Continuously red-teams agents across tools, MCP servers and workflows

What this layer will not do

Testing finds weaknesses; it does not block them in production. A clean scan covers only the probes it ran, and OWASP's LLM01 page says it is unclear whether fool-proof prevention of prompt injection exists, so a pass shows only that those probes failed that day. Results go stale with every model, prompt or tool change. Benchmarks compare models on fixed datasets and do not replace tests against your own app, data and tools. Every finding needs a fix in another layer: a guardrail, a data control or a narrower agent permission.

Related guides

Testing and red teaming FAQ

How often should we red team an LLM app?

The OWASP LLM01 page lists adversarial testing and attack simulations as a prevention step without setting an interval, and the OWASP Agentic Top 10 asks for periodic red-team tests of goal override (ASI01). Our analysis: run automated scans in CI on every model, prompt or tool change, and schedule deeper expert or managed exercises for high-risk apps and agents.

What is the difference between an evaluation framework and a red teaming tool?

Our analysis: an evaluation framework measures how a model performs on a set of tasks or benchmarks and often sits with ML teams. A red teaming tool generates attacks such as injection, jailbreaks and data extraction against an app or agent and reports what got through. Both belong in this layer; each tool page says which jobs its sources document.

Which NIST AI RMF function does testing answer?

MEASURE. NIST describes it as using testing and assessment methods, and its subcategories include independent assessors (MEASURE 1.3), evaluating and documenting security and resilience (MEASURE 2.7) and evaluating fairness and bias (MEASURE 2.11). GOVERN 4.3 adds organizational practices that enable AI testing.

Framework sources & last reviewed

Last reviewed October 21, 2026. We compile from the vendors' own public pages and repositories and we do not run the products. Source links open in a new tab. Found a mistake? Report a correction.

Drafted with AI assistance; facts checked against primary sources on October 21, 2026.

OWASP Top 10 for LLM Applications 2025 (list page) (1 sources)