Promptfoo Review 2026: Pricing, Features, Pros & Cons
Promptfoo makes LLM evaluation a YAML file that fails your build, not a dashboard someone set up in March and never opened again. Here's an honest look at what the free CLI covers, where the red-teaming suite earns its place, and the thing it deliberately refuses to do in 2026.
Quick Verdict
Best for: engineering teams who want prompt regression tests in CI, honest side-by-side model comparisons, and a repeatable adversarial scan before a customer-facing LLM feature ships. Skip it if: what you actually need is production observability — that is a different tool and Promptfoo does not pretend otherwise.
What Is Promptfoo?
Promptfoo is an open-source command-line tool for testing LLM prompts and applications. You write a config file listing your prompts, the providers to run them against, a set of test cases with variables, and assertions describing what a correct output looks like. Running the CLI executes every combination and prints a matrix; running it in CI turns a regression into a non-zero exit code and a failed check on the pull request.
Assertions are the part that determines whether an evaluation suite is useful or theatrical, and the range here is good: exact and substring matching, similarity thresholds, JSON schema validation, arbitrary Python or JavaScript functions for deterministic checks, and LLM-as-judge rubrics for the cases where nothing cheaper will do. The tool nudges you toward the deterministic options first, which is the correct instinct — a graded assertion you can trust beats a graded assertion that sounds sophisticated.
The second half of the product is red teaming, and it is a more serious piece of engineering than the category usually ships in open source. Given a description of your application, it generates adversarial inputs across attack classes — prompt injection, jailbreaking, system-prompt extraction, PII leakage, unsafe output — runs them, and reports what got through. For a team about to expose an LLM to the public with no security budget, that is a meaningful pre-launch artefact rather than a checkbox.
Built an LLM evaluation, testing or AI security tool? People land on this review while they are still choosing one.
Add it to the coding category — a free listing publishes after review, and it is the same page ChatGPT, Perplexity and Google read when a developer asks which LLM eval tool to use. Want it live in minutes with a Verified badge instead? That option is on the form, one-time, no subscription.
Promptfoo Pros & Cons
✓ Pros
- •Evaluation is a file in your repo, not a dashboard someone configured once and left — the config is reviewed in a pull request and diffed like any other code
- •Genuinely fast to adopt: a useful eval exists within an hour of installing the CLI, which is not true of most tools in this category
- •Runs locally and in CI with no account, no data leaving your machine and no vendor in the request path — a decisive advantage for teams whose prompts contain things legal cares about
- •Model comparison is the default view rather than a feature: point the same test set at four providers and read the side-by-side, which makes model-selection arguments empirical
- •The red-teaming suite is unusually serious for an open-source tool, generating adversarial cases for prompt injection, jailbreaks, PII leakage and unsafe output rather than just running your own test cases
- •Assertion types cover the range that matters — exact and fuzzy matching, JSON schema validation, deterministic Python or JavaScript checks, and LLM-as-judge when nothing cheaper will do
✗ Cons
- •The YAML grows teeth: past a few dozen test cases with shared providers and assertion sets, the config becomes its own maintenance surface and people start generating it programmatically
- •It evaluates prompts and outputs well, but it is not a tracing tool — it will not show you what a live agent did in production, which is a different product category you will still need
- •Multi-step agent and tool-use evaluation is workable but clearly less native than single-prompt evaluation, which is where the design started
- •The local web viewer is functional rather than polished, and teams wanting shareable dashboards, run history and collaboration are being pointed at the commercial tier
- •LLM-as-judge assertions inherit all the usual unreliability, and it is easy to build a confident-looking suite whose grader is the weakest component in it
- •Enterprise pricing is quote-based with no public number, so the cost of the collaborative path cannot be evaluated without a sales conversation
Promptfoo Pricing 2026
The open-core split here is cleaner than most: the thing that does the work is free and complete, and the paid tiers sell collaboration, hosting and compliance rather than capability. Nothing in the evaluation or red-teaming engine is held back for the enterprise tier.
Open Source
- •Full CLI and eval engine
- •All assertion types
- •Red-teaming suite
- •Local web viewer
Individuals and teams running evals in their own CI
Cloud
- •Hosted run history
- •Team sharing and comparison
- •Managed red-team scans
- •Access controls
Teams who want shared results without hosting anything
Enterprise
- •Self-hosted deployment
- •SSO and audit logging
- •Compliance reporting
- •Vendor support and SLAs
Regulated organisations that need red-team evidence on file
Tier structure as publicly documented at promptfoo.dev as of September 2026. Commercial tiers are quote-based rather than self-serve, so no figure is published. Your evaluation runs still cost model tokens, and a large matrix costs more of them than people expect.
Promptfoo vs Braintrust vs LangSmith
| Feature | Promptfoo | Braintrust | LangSmith |
|---|---|---|---|
| Config in your repo | ✅ YAML, reviewed in PRs | ⚠️ SDK plus dashboard | ⚠️ SDK plus dashboard |
| Works with no account | ✅ Fully local | ❌ Hosted-first | ❌ Hosted-first |
| Production tracing | ❌ Not its job | ✅ Logs and traces | ✅ Deep tracing |
| Red teaming | ✅ Built-in suite | ❌ Not included | ❌ Not included |
| Side-by-side model comparison | ✅ Default view | ✅ Experiments | ✅ Experiments |
| Dataset management | ⚠️ Files you manage | ✅ First-class | ✅ First-class |
| CI integration | ✅ Exit code, no service | ✅ Via SDK | ✅ Via SDK |
| Public pricing | ⚠️ OSS free, enterprise quoted | ⚠️ Partly published | ✅ Published tiers |
When Promptfoo Is the Right Call
The strongest case is a team that already has CI discipline and has just started shipping LLM features into it. Everything about the tool is shaped to fit that existing habit: the config is a file, the run is a command, the failure is an exit code, and the review happens in a pull request alongside the change that caused it. Adoption cost is close to zero because nothing new has to be introduced organisationally — no dashboard to own, no account to provision, no data-processing agreement to negotiate.
The second case is model selection under pressure. When a new model ships and someone asks whether to switch, the difference between an opinion and an answer is a test set you can point at both. Promptfoo makes that a config change and a coffee break, and the side-by-side output is legible enough to settle the argument in a meeting.
Where it is the wrong call is when the real problem is production, not pre-production. If your question is why a specific customer got a bad answer on Tuesday, no offline eval suite will tell you — you need traces, and Promptfoo has none by design. It is also a poor fit for teams who want non-engineers curating datasets and reading results, because the workflow is a repository, and a repository is a wall to most of the people who would otherwise contribute test cases.
Frequently Asked Questions
Is Promptfoo free?
The open-source CLI is free under an MIT licence and it is not a demo version — the full evaluation engine, every assertion type and the red-teaming suite all run locally with no account and no key beyond the model provider credentials you already have. The commercial products sit above it: a cloud tier for shared run history and team collaboration, and an enterprise tier for self-hosted deployment, SSO and compliance reporting. A large number of teams never leave the free tier, and nothing in the tool nags them to.
Promptfoo vs Braintrust — which should I use?
Braintrust is a platform: datasets, experiments, production logging and evaluation in one hosted place, aimed at teams who want the whole loop managed. Promptfoo is a tool: a config file and a CLI that answers 'did this prompt change make things worse' in CI, and stops there. If you want one system of record for everything an LLM does in your product, Braintrust is the shape of that. If you want prompt regression tests that live in the repo and block a bad pull request, Promptfoo does that with far less adoption cost — and plenty of teams run both, because they are answering different questions.
Promptfoo vs LangSmith — what's the difference?
LangSmith's centre of gravity is tracing: seeing exactly what a running chain or agent did, step by step, in production, with evaluation built around those traces. Promptfoo has no tracing at all and does not want any — it is an offline testing tool. The practical split is that LangSmith answers 'what happened in that run last night' and Promptfoo answers 'will this prompt change break anything'. LangSmith also pulls hardest for teams already inside the LangChain ecosystem, whereas Promptfoo is deliberately framework-agnostic.
What does the red-teaming feature actually do?
You point it at your application or prompt and it generates adversarial test cases across a set of attack categories — prompt injection, jailbreaks, attempts to extract the system prompt, PII leakage, harmful-content elicitation, and role-confusion attacks — then runs them and reports which ones got through. It is not a substitute for a security review by a person, and it will produce some noise, but as a repeatable pre-launch check on a customer-facing LLM feature it is considerably better than the alternative most teams have, which is nothing.
Can I use Promptfoo without sending data to a third party?
Yes, and this is one of its stronger selling points. The CLI runs entirely on your machine or your CI runner; the only outbound calls are to whichever model provider you are testing against, and those go direct. There is no Promptfoo service in the request path, no telemetry requirement, and results are written to local files. For teams whose prompts contain customer data or unreleased product detail, that removes the review that usually stalls adoption of a hosted evaluation platform.
Does Promptfoo work for agents, not just single prompts?
It can, with more assembly. The tool was designed around evaluating a prompt or a provider call, and that path is smooth; evaluating a multi-step agent means wrapping your agent as a custom provider so Promptfoo can call it, then asserting on the final output or on intermediate state you expose. That works and teams do it, but you are building the harness. If agent-level evaluation is the main thing you need rather than a side case, look at tools that took multi-step traces as their starting assumption.
Compare AI Evaluation Tools
See how Promptfoo stacks up against Braintrust, LangGraph and every other AI development tool in the directory.
Affiliate disclosure: Some links on this page are affiliate links. If you sign up through them, AISO Tools may earn a commission at no extra cost to you. This never affects our rankings or reviews.
📬 Get the best new AI tools delivered weekly
One concise email with fresh launches, trending picks, and featured standouts.
Join thousands of professionals who discover the best AI tools every week. No spam — unsubscribe anytime.