AI Defense Stack

Comparison: Red teaming

garak vs PyRIT

Choosing between garak and PyRIT decides whether your LLM red teaming starts from a ready-made probe library on the command line or from attack code your team writes in Python. Both are free, open source and ship multi-turn attacks. garak picks detectors for each probe and writes report and hit logs; PyRIT adds configurable scorers, a results database, a local web app for human-led testing and Playwright targets for web apps.

  • Last reviewed

Executive summary

What it is
garak and PyRIT are free, open-source red teaming tools that attack an LLM or LLM app on purpose to find weaknesses before release. garak fits a quick scan from the command line; PyRIT fits custom attack campaigns written in Python.
The risk
Models and apps that ship without adversarial testing can follow injected instructions, leak data or produce harmful output in production.
What to buy or do
Run garak for a fast probe sweep of a model or REST endpoint. Adopt PyRIT when your team will script multi-turn campaigns, score results, add human-led testing or reach web apps through Playwright. Running both is reasonable.

Which should you choose?

Our analysis, built from the facts in the table below.

Choose garak if

Our analysis
  • You want a ready-made probe library that you run against a model or endpoint with one python3 -m garak command.
  • You want the scanner to find appropriate detectors and handle rate limiting for you, as its docs say it does.
  • You want probes for prompt injection, jailbreaks, data leakage, misinformation and toxicity listed in one README.

See garak.

Choose PyRIT if

Our analysis
  • You want to write Python that builds and chains your own attack runs, including named multi-turn strategies such as Crescendo and Skeleton Key.
  • You want CoPyRIT, a local web app for human-led red teaming, next to automated runs.
  • You need to test web apps through a browser with Playwright, which the PyRIT docs list as a target type.

See PyRIT.

Side by side

Every cell links to the page that states it, with the date we checked it. We compile from public pages and repositories and do not run either product. Coverage Score is computed within one category from sourced facts (how it works).

garak compared with PyRIT: each row shows the fact for both tools with its source
Attribute garak PyRIT
Vendor NVIDIA Source, checked October 21, 2026: github.com (opens in a new tab) Microsoft Source, checked October 21, 2026: github.com (opens in a new tab)
Status Active Source, checked October 21, 2026: api.github.com (opens in a new tab) Active Source, checked October 21, 2026: api.github.com (opens in a new tab)
Open source and license Open source, Apache-2.0 Source, checked October 21, 2026: github.com (opens in a new tab) Open source, MIT Source, checked October 21, 2026: github.com (opens in a new tab)
GitHub stars 9,636 GitHub stars as of Oct 21, 2026; last push Oct 21, 2026 Source, checked October 21, 2026: api.github.com (opens in a new tab) 4,668 GitHub stars as of Oct 21, 2026; last push Oct 22, 2026 Source, checked October 21, 2026: api.github.com (opens in a new tab)
Latest release October 16, 2026 Source, checked October 21, 2026: api.github.com (opens in a new tab) September 4, 2026 Source, checked October 21, 2026: api.github.com (opens in a new tab)
Deployment CLI, Library, Self-hosted Source, checked October 21, 2026: docs.garak.ai (opens in a new tab) Library, CLI, Self-hosted Source, checked October 21, 2026: microsoft.github.io (opens in a new tab)
Pricing model Free Source, checked October 21, 2026: github.com (opens in a new tab) Free Source, checked October 21, 2026: github.com (opens in a new tab)
OWASP LLM risks Our analysis
  • LLM01 Prompt Injection
  • LLM02 Sensitive Information Disclosure
  • LLM09 Misinformation
  • LLM01 Prompt Injection
  • LLM02 Sensitive Information Disclosure
Documented capabilities
  • Probes for prompt injection
  • Probes for jailbreaks
  • Probes for data leakage
  • Probes for hallucination and misinformation
  • Probes for toxicity generation
  • Runs its probes without supervision, choosing detectors and handling rate limiting itself
  • Writes screen output, a report log, a hit log and a debug log
  • Includes the GOAT probe, a multi-turn jailbreak that uses an attacker LLM to craft adversarial prompts across several turns
  • Includes TAP and PAIR probes that use an LLM to refine attack prompts iteratively
Source, checked October 21, 2026: github.com (opens in a new tab) Source, checked October 21, 2026: docs.garak.ai (opens in a new tab) Source, checked October 21, 2026: github.com (opens in a new tab) Source, checked October 21, 2026: github.com (opens in a new tab)
  • Runs multi-turn attack strategies including Crescendo, TAP and Skeleton Key
  • Automates single-turn and multi-turn attacks against a target
  • Includes standardized scenarios that cover data leakage, content harms and psychosocial risks
  • Provides the pyrit_scan command-line scanner, documented for automated single-command runs, and an interactive pyrit_shell
  • API reference lists dataset and benchmark modules alongside attack, scorer and target modules
  • Includes a Jailbreak scenario that tests how vulnerable a model is to jailbreak templates
Source, checked October 21, 2026: microsoft.github.io (opens in a new tab) Source, checked October 21, 2026: microsoft.github.io (opens in a new tab) Source, checked October 21, 2026: microsoft.github.io (opens in a new tab) Source, checked October 21, 2026: microsoft.github.io (opens in a new tab)
Integrations 4 listed
  • Hugging Face Hub models
  • OpenAI API models
  • AWS Bedrock models
  • LiteLLM
Source, checked October 21, 2026: github.com (opens in a new tab) 		3 listed
  • OpenAI targets
  • Anthropic targets
  • Hugging Face targets
Source, checked October 21, 2026: microsoft.github.io (opens in a new tab)
Documentation
  • Documentation site
  • API reference
  • Quickstart
Source, checked October 21, 2026: docs.garak.ai (opens in a new tab) Source, checked October 21, 2026: reference.garak.ai (opens in a new tab) Source, checked October 21, 2026: docs.garak.ai (opens in a new tab)
  • Documentation site
  • API reference
  • Quickstart
Source, checked October 21, 2026: microsoft.github.io (opens in a new tab) Source, checked October 21, 2026: microsoft.github.io (opens in a new tab) Source, checked October 21, 2026: microsoft.github.io (opens in a new tab)
How you run it Command line: python3 -m garak with a target type, a target name and a probe spec Source, checked October 21, 2026: github.com (opens in a new tab) Python API, the pyrit_scan command, an interactive pyrit_shell and the CoPyRIT local web app Source, checked October 21, 2026: microsoft.github.io (opens in a new tab)
Building blocks Probes, detectors, generators, harnesses and evaluators Source, checked October 21, 2026: github.com (opens in a new tab) Targets, attacks, scorers (true/false, Likert scale, classification) and built-in memory Source, checked October 21, 2026: microsoft.github.io (opens in a new tab)
Where results go Screen output, a report log, a hit log and a debug log Source, checked October 21, 2026: docs.garak.ai (opens in a new tab) Conversations, scores and attack results stored in SQLite or Azure SQL Source, checked October 21, 2026: microsoft.github.io (opens in a new tab)
How you install it python -m pip install -U garak Source, checked October 21, 2026: github.com (opens in a new tab) With pip or uv, or as a pre-configured Docker container with JupyterLab Source, checked October 21, 2026: microsoft.github.io (opens in a new tab)
Multi-turn attacks GOAT probe: a multi-turn jailbreak that uses an attacker LLM to craft adversarial prompts across several turns Source, checked October 21, 2026: github.com (opens in a new tab) Multi-turn attack strategies such as Crescendo, TAP and Skeleton Key, alongside single-turn attacks Source, checked October 21, 2026: microsoft.github.io (opens in a new tab)
TAP attacks TAP and PAIR probes that use an LLM to refine attack prompts iteratively (TAP prunes prompts unlikely to jailbreak the target) Source, checked October 21, 2026: github.com (opens in a new tab) TAP is one of the multi-turn strategies named on the docs home page Source, checked October 21, 2026: microsoft.github.io (opens in a new tab)
REST and HTTP endpoints A REST generator for any endpoint that returns plaintext or JSON, which the README describes as pretty much anything accessible via REST Source, checked October 21, 2026: github.com (opens in a new tab) Custom HTTP endpoints, listed next to OpenAI, Azure, Anthropic, Google and Hugging Face targets Source, checked October 21, 2026: microsoft.github.io (opens in a new tab)
WebSocket and browser targets A REST generator (rest.py) and a WebSocket generator (websocket.py) among 26 files in the generators directory, and no browser-driven generator Source, checked October 21, 2026: github.com (opens in a new tab) Custom WebSocket endpoints, and web app targets driven through Playwright Source, checked October 21, 2026: microsoft.github.io (opens in a new tab)
Coverage Score 82 out of 100 Computed from sourced facts. What this measures 73 out of 100 Computed from sourced facts. What this measures
Where each is stronger Our analysis
  • Command-line first, with a built-in probe library: the README's scan examples are single python3 -m garak commands that choose a target and a probe spec.
  • Lists probes for hallucination, data leakage, prompt injection, misinformation, toxicity generation and jailbreaks in its README.
  • Its docs say it finds appropriate detectors and handles rate limiting during a run. PyRIT's pyrit_scan also runs from one command, so the difference is how much garak sets up for you.
  • Is a framework for building your own attack runs in Python, with targets, attacks, scorers (true/false, Likert, classification) and built-in memory, plus named multi-turn strategies such as Crescendo and Skeleton Key.
  • Offers a graphical local app for human-led red teaming and stores conversations, scores and results in SQLite or Azure SQL.
  • Documents web app targets driven through Playwright; the garak generators directory we checked lists REST and WebSocket generators but no browser-driven one.

Verdict

Our analysis

Our analysis

Use garak when you want a scan you can start with one command: point it at a model or REST endpoint and read a log of which probes got through. Use PyRIT when you want to build and tune attack runs in Python, mix automated runs with human-led testing in CoPyRIT, or reach web apps through Playwright. Both are free, both ship multi-turn attacks such as TAP and both reach custom HTTP and WebSocket endpoints, so running both is a reasonable plan. We compare documented features only and did not measure how well either finds weaknesses; run both against your own application.

garak vs PyRIT: FAQ

Are garak and PyRIT free?

Yes. garak is released under the Apache-2.0 license and PyRIT under the MIT License. We found no paid tier in the README or docs of either project.

What is the main difference between garak and PyRIT?

garak is a command-line scanner with a library of probes and detectors that logs which attempts succeeded. PyRIT is a Python framework for building automated and human-led attacks, with scorers and built-in memory. The table above shows the source for each fact.

Who makes garak and PyRIT?

garak is published in NVIDIA's GitHub organization and PyRIT in Microsoft's. PyRIT's documentation describes it as a framework for assessing the security and safety of generative AI systems at scale.

Can both tools run from the command line?

Yes. garak runs as python3 -m garak with a target and a probe spec. PyRIT documents a pyrit_scan command and an interactive pyrit_shell, and it can also be used as a Python library.

Do both tools support multi-turn attacks?

Yes. garak includes the GOAT probe, a multi-turn jailbreak, and TAP and PAIR probes that refine attack prompts iteratively. PyRIT documents multi-turn strategies such as Crescendo, TAP and Skeleton Key. We did not compare how well either works.

Key facts you can cite

Keep exploring

Sources & last reviewed

Last reviewed . We compile from the vendors' own public pages and repositories and we do not run the products. Source links open in a new tab. Found a mistake? Report a correction.

Drafted with AI assistance; facts checked against primary sources on October 21, 2026.

garak (12 sources)
PyRIT (10 sources)
Comparison rows (16 sources)