AI Defense Stack

Comparison: Red teaming

promptfoo vs garak

To red team a deployed LLM application on every pull request, Promptfoo fits the job; for a quick probe scan of a model or REST endpoint from the command line, garak does. Both are open source and test for jailbreaks and prompt injection. Promptfoo is a Node.js CLI and library driven by a config file that also runs evals; garak is a Python package that runs a library of probes from command-line flags.

  • Last reviewed

Executive summary

What it is
Promptfoo and garak are open-source tools that attack LLMs on purpose to find weaknesses such as jailbreaks and prompt injection. Promptfoo tests whole applications and runs in CI; garak scans models and endpoints from the command line.
The risk
Untested LLM apps can follow injected instructions, leak data or produce harmful output in production. Testing only the bare model misses weaknesses added by your prompts, retrieval and tools, so the test target is part of the decision.
What to buy or do
Choose Promptfoo to red team an endpoint, code or browser flow from a config file and run it on pull requests with its GitHub Action. Choose garak to probe a model or endpoint with no config file. Check Promptfoo's paid plans and ownership first.

Which should you choose?

Our analysis, built from the facts in the table below.

Choose Promptfoo if

Our analysis
  • You want to red team a whole application through an HTTP endpoint, custom code or a browser, and not only a bare model.
  • You want evals and red teaming in one tool, with the documented GitHub Action running on pull requests.
  • You want a report view with failure cases, a test breakdown and the raw inputs and outputs.

See Promptfoo.

Choose garak if

Our analysis
  • You want a plain Python package that runs from command-line flags, with no config file and no Node.js.
  • You want a fixed library of probes for hallucination, data leakage, prompt injection, misinformation, toxicity and jailbreaks.
  • You want a hit log that records each vulnerability found, next to a report log of every prompt and response.

See garak.

Side by side

Every cell links to the page that states it, with the date we checked it. We compile from public pages and repositories and do not run either product. Coverage Score is computed within one category from sourced facts (how it works).

Promptfoo compared with garak: each row shows the fact for both tools with its source
Attribute Promptfoo garak
Vendor OpenAI (Promptfoo) Source, checked October 21, 2026: developers.openai.com (opens in a new tab) NVIDIA Source, checked October 21, 2026: github.com (opens in a new tab)
Status Active Source, checked October 21, 2026: api.github.com (opens in a new tab) Active Source, checked October 21, 2026: api.github.com (opens in a new tab)
Ownership Promptfoo announced on March 9, 2026 that it had agreed to be acquired by OpenAI. OpenAI's developer documentation now says Promptfoo is part of OpenAI; we found no published closing date. Source, checked October 21, 2026: promptfoo.dev (opens in a new tab) No ownership change documented in the sources we checked
Open source and license Open source, MIT Source, checked October 21, 2026: github.com (opens in a new tab) Open source, Apache-2.0 Source, checked October 21, 2026: github.com (opens in a new tab)
GitHub stars 26,250 GitHub stars as of Oct 21, 2026; last push Oct 22, 2026 Source, checked October 21, 2026: api.github.com (opens in a new tab) 9,636 GitHub stars as of Oct 21, 2026; last push Oct 21, 2026 Source, checked October 21, 2026: api.github.com (opens in a new tab)
Latest release October 8, 2026 Source, checked October 21, 2026: api.github.com (opens in a new tab) October 16, 2026 Source, checked October 21, 2026: api.github.com (opens in a new tab)
Deployment CLI, Library, Self-hosted, SaaS, On-prem Source, checked October 21, 2026: promptfoo.dev (opens in a new tab) CLI, Library, Self-hosted Source, checked October 21, 2026: docs.garak.ai (opens in a new tab)
Pricing model Freemium: Community: Free Forever, with red teaming at 10k probes per month. Enterprise: Custom, with managed cloud deployment. On-Premise: Custom, deployed on your own infrastructure. (as listed on October 21, 2026; check the vendor for current pricing) Source, checked October 21, 2026: promptfoo.dev (opens in a new tab) Free Source, checked October 21, 2026: github.com (opens in a new tab)
OWASP LLM risks Our analysis
  • LLM01 Prompt Injection
  • LLM02 Sensitive Information Disclosure
  • LLM09 Misinformation
  • LLM01 Prompt Injection
  • LLM02 Sensitive Information Disclosure
  • LLM09 Misinformation
Documented capabilities
  • Scans for jailbreaks among more than 50 vulnerability types
  • Scans for injections among more than 50 vulnerability types
  • Scans a target automatically across many vulnerability types from one command
  • Generates adversarial inputs with plugins and wraps them in attack strategies
  • Supports custom policies that enforce an organization's own guidelines
  • Offers red team presets for the OWASP LLM Top 10, NIST AI RMF and MITRE ATLAS, and lists EU AI Act and ISO 42001 as framework filters
  • Provides plugins that test for PII leakage: direct, across sessions, through social engineering and through API or database access
  • Provides hallucination and overreliance plugins that test for misinformation
  • Produces a report with test types, severity levels and suggested mitigations
  • Runs evals on pull requests through a documented GitHub Action
Source, checked October 21, 2026: promptfoo.dev (opens in a new tab) Source, checked October 21, 2026: promptfoo.dev (opens in a new tab) Source, checked October 21, 2026: promptfoo.dev (opens in a new tab) Source, checked October 21, 2026: promptfoo.dev (opens in a new tab)
  • Probes for prompt injection
  • Probes for jailbreaks
  • Probes for data leakage
  • Probes for hallucination and misinformation
  • Probes for toxicity generation
  • Runs its probes without supervision, choosing detectors and handling rate limiting itself
  • Writes screen output, a report log, a hit log and a debug log
  • Includes the GOAT probe, a multi-turn jailbreak that uses an attacker LLM to craft adversarial prompts across several turns
  • Includes TAP and PAIR probes that use an LLM to refine attack prompts iteratively
Source, checked October 21, 2026: github.com (opens in a new tab) Source, checked October 21, 2026: docs.garak.ai (opens in a new tab) Source, checked October 21, 2026: github.com (opens in a new tab) Source, checked October 21, 2026: github.com (opens in a new tab)
Integrations 5 listed
  • OpenAI models
  • Anthropic models
  • AWS Bedrock models
  • Hugging Face models
  • GitHub Actions
Source, checked October 21, 2026: github.com (opens in a new tab) Source, checked October 21, 2026: promptfoo.dev (opens in a new tab) Source, checked October 21, 2026: promptfoo.dev (opens in a new tab) 		4 listed
  • Hugging Face Hub models
  • OpenAI API models
  • AWS Bedrock models
  • LiteLLM
Source, checked October 21, 2026: github.com (opens in a new tab)
Documentation
  • Documentation site
  • Quickstart
Source, checked October 21, 2026: promptfoo.dev (opens in a new tab) Source, checked October 21, 2026: promptfoo.dev (opens in a new tab)
  • Documentation site
  • API reference
  • Quickstart
Source, checked October 21, 2026: docs.garak.ai (opens in a new tab) Source, checked October 21, 2026: reference.garak.ai (opens in a new tab) Source, checked October 21, 2026: docs.garak.ai (opens in a new tab)
How you configure a run A promptfooconfig.yaml file that defines the target, then promptfoo redteam run Source, checked October 21, 2026: promptfoo.dev (opens in a new tab) Command-line flags for target type, target name and probe spec, with python3 -m garak Source, checked October 21, 2026: github.com (opens in a new tab)
What you can point it at HTTP endpoints, direct models, custom Python or JavaScript code and a browser provider Source, checked October 21, 2026: promptfoo.dev (opens in a new tab) Model generators such as Hugging Face Hub, OpenAI, Cohere, AWS Bedrock, LiteLLM and anything reachable over REST Source, checked October 21, 2026: github.com (opens in a new tab)
How attacks are produced Plugins generate adversarial inputs and strategies wrap them in attack patterns; the run writes several hundred inputs to redteam.yaml Source, checked October 21, 2026: promptfoo.dev (opens in a new tab) Probes generate interactions with the model and detectors identify when it fails Source, checked October 21, 2026: github.com (opens in a new tab)
What you need installed Node.js 22.22.0 or newer, then npx promptfoo@latest redteam setup Source, checked October 21, 2026: promptfoo.dev (opens in a new tab) A Python environment, then python -m pip install -U garak Source, checked October 21, 2026: github.com (opens in a new tab)
Coverage Score 80 out of 100 Computed from sourced facts. What this measures 82 out of 100 Computed from sourced facts. What this measures
Where each is stronger Our analysis
  • Targets applications: HTTP endpoints, direct models, Python or JavaScript code and a browser provider, not only model connectors.
  • Documents evals as well as red teaming, and a GitHub Action for pull requests; we found no CI integration in the garak sources we checked.
  • Documents plugins that generate adversarial inputs and strategies that wrap them in attack patterns, and documents red team presets for the OWASP LLM Top 10, NIST AI RMF and MITRE ATLAS.
  • Installs as a Python package and runs from command-line flags, so a first scan needs no config file.
  • Lists no paid tier in its README, while Promptfoo's pricing page lists Enterprise and On-Premise plans with custom pricing.

Verdict

Our analysis

Our analysis

Choose Promptfoo when you test an application or endpoint, keep the test config in your repository and want evals and red teaming to run on pull requests. Choose garak when you want a quick scan of a model or REST endpoint with a fixed probe library and no config file. Both are open source and test for jailbreaks and prompt injection. Promptfoo also sells Enterprise and On-Premise plans, and OpenAI's documentation says Promptfoo is part of OpenAI, so check the repository for how the project continues. We compare documented features only and did not measure detection results; run both on your own system.

Promptfoo vs garak: FAQ

Are promptfoo and garak open source?

Yes. Promptfoo is released under the MIT License and garak under the Apache-2.0 license, and both code bases are on GitHub. Promptfoo also sells Enterprise and On-Premise plans with custom pricing, while its Community plan is free.

What is the main difference between promptfoo and garak?

Promptfoo is configured with a YAML file, can target HTTP endpoints, code and a browser, and also runs evals. garak runs from command-line flags against model generators with a library of probes and detectors. The table above shows the source for each fact.

Can promptfoo or garak run in CI?

Promptfoo documents a GitHub Action that runs evals on pull requests. The garak README we checked does not mention CI integration.

Who owns Promptfoo?

According to OpenAI's developer documentation, Promptfoo is part of OpenAI. Promptfoo announced on March 9, 2026 that it had agreed to be acquired by OpenAI, and we found no page that gives a closing date.

Key facts you can cite

Keep exploring

Sources & last reviewed

Last reviewed . We compile from the vendors' own public pages and repositories and we do not run the products. Source links open in a new tab. Found a mistake? Report a correction.

Drafted with AI assistance; facts checked against primary sources on October 21, 2026.

Promptfoo (14 sources)
garak (12 sources)
Comparison rows (8 sources)