Answers
Microsoft's AI Red Teaming Agent docs show a model refusing a direct harmful request and answering it once the characters are flipped. AI red teaming hunts for that kind of failure before release; it finds risks, and systematic measurement then tells you how common they are.
AI red teaming is probing an AI system the way an adversarial user would, to find failures before release. Microsoft says the term now covers novel content risks as well as security risks, and describes it as simulating a user who tries to make the system misbehave. Its planning guide says the goal is to identify harms and map the risk surface, which tells you what to measure and mitigate next.
A penetration test looks for security vulnerabilities; AI red teaming also probes for harmful or unintended model behavior that even ordinary use can trigger. Microsoft contrasts traditional red teaming, which exploits the cyber kill chain, with AI red teaming, which probes novel content and security risks. Its planning guide adds that benign as well as adversarial use of an LLM can produce harmful output.
Common choices include PyRIT, garak and promptfoo, alongside commercial platforms and managed services. PyRIT supports automated and human-led red teaming with multi-turn strategies such as Crescendo and TAP. garak probes for prompt injection, jailbreaks and guardrail bypass, and promptfoo documents red team scans in CI pipelines. Our category page compares the listed tools by license, deployment and OWASP coverage.
No. Automated scans repeat known attack patterns quickly, but Microsoft says the most effective risk assessment combines automated tools with expert human analysis. Its planning guide advises an initial round of manual red teaming before systematic measurement, and its AI Red Teaming Agent page warns that model-based scoring can produce false positives, so review results before acting on them.
Tie it to change, because no standard we found sets a fixed interval. Microsoft's planning guide describes iterative rounds across the product life cycle, and its AI Red Teaming Agent page suggests automated scans during design, development and pre-deployment, plus scheduled runs after deployment. In our analysis, rerun whenever the model, system prompt, tools or reachable data sources change.
List the top issues, link the raw data and preview the next round, as Microsoft's planning guide suggests. For each finding, record the date, a unique ID for reproduction, the input prompt and a description of the output, and state which endpoints were used. Say plainly that red teaming identifies risk and does not measure how common it is. Automated tools add attack success rate by technique and risk category.
Attack success rate (ASR) is the share of attack attempts that produced the unwanted result, out of all attempts. Microsoft's AI Red Teaming Agent reports it by attack technique and risk category. Use it to compare runs, not as a safety score: the agent scores responses with generative models, which can be non-deterministic and produce false positives, and its synthetic data does not reflect real-world distributions.
Three public references are common starting points. The OWASP GenAI Red Teaming Guide covers the model, its implementation, the supporting infrastructure and runtime behavior. MITRE ATLAS is a public knowledge base of adversary tactics, techniques and procedures against AI systems, and the voluntary NIST AI RMF with its Generative AI Profile frames the risks. We map tools to the OWASP LLM Top 10 and to ATLAS.
Yes, for repeatable automated scans. promptfoo's documentation shows a red team scan in a daily scheduled GitHub Actions workflow, gives GitLab CI and Jenkins examples, and shows builds that fail below a quality threshold. Microsoft's AI Red Teaming Agent also describes scheduled runs after deployment. In our analysis, treat CI runs as regression checks for known attacks and keep human-led rounds for finding new ones.