SuperRed Open-source AI red-teaming and AI system security framework

SuperRed

Run any attacker against any AI system with any threat model

or and Get started

SuperRed is an open-source framework for red-teaming AI systems (chatbots, agents, assistants): deliberately attacking them to see which security properties can be violated. It keeps the attacker, the system under test, and the benchmark that scores the outcome as separate, interchangeable modules.
The framework allows red-teamers to implement powerful, reusable strategies, system builders to easily test systems, and evaluators to measure any combination of attack and target. Each run is governed by a fine-grained threat model: the attacker's model and per-task budget, the scope of trust boundaries it may observe and control, and whether it receives the benchmark's feedback. The framework provides ready-to-use metrics and result dashboards.

Why SuperRed

Attacks, systems, and benchmarks, freely composable

Module · Attacker

Optimizer

Drives the attack.

Framework

Controller

Orchestrates the run and enforces the security scope.

Module · System

Target

The AI system under test.

Module · Benchmark

Security claim

Defines the goal, scores the run.

Attacks and benchmarks live in one-off codebases, each wired to the system it was first demonstrated on. SuperRed pulls the pieces apart: the optimizer, target, and security claim are interchangeable modules that meet only at the controller. Swap any one and the comparison becomes a change of a single argument, not a re-implementation project.

Showcase

Ready-made attackers, targets, and benchmarks, plus the research built on them

Attacks, targets, and benchmarks, ready to run

Over a dozen faithfully implemented attacks, including AutoDAN-Turbo, PAIR, TAP, Crescendo, and AgentVigil, alongside targets and safety benchmarks such as HarmBench, DecodingTrust, and AgentDojo.

Browse the modules →

Research built with SuperRed

From new attack strategies to full system evaluations, SuperRed is the substrate researchers and builders use to do serious red-teaming work.

See the research →

A meeting point

Attackers, targets, and benchmarks are all first-class

Attack, system, and benchmark are the same kind of thing here: interchangeable modules. Hold any one fixed to evaluate it; every contribution strengthens a shared toolbox.

For red-teamers

An interactive environment for developing attacks, so you can build and score novel strategies on real environments as you go.

  • Powerful attackers of every kind: deterministic to agentic, fixed strategies to RL.
  • Iterate against many targets as you build, and evaluate on standard benchmarks in a few lines.
Write an optimizer →

For system builders

Declare your system's trust boundaries once, then see exactly which attacks succeed at each level of access.

  • Test any kind of target: fixed-response fixtures, simulated environments, sandboxes, or live deployments.
  • Wrap an existing system as a target; porting is straightforward.
Write a target →

For evaluators

Package a benchmark once as a security claim, then score any attacker against any target, or reuse the ones already ported.

  • Benchmark under many threat models for fine-grained, comparable insight, not assumptions that vary from benchmark to benchmark.
  • Score deterministically or agentically, whichever the benchmark calls for.
Write a task →

High-level features

What the framework gives you

Flexible threat models

Switch on exactly the capabilities the attacker gets, from a single injected prompt up to full compromise, and change them between runs.

Fair and reproducible

A fixed model and a per-task cost budget bound every attacker equally.

One threat model (scope, llm, budget)
Scope the system's trust boundaries 
system
├─ system_prompt
├─ model_responses
└─ architecture
   └─ high_level_architecture
user
…

the trust boundaries the attacker may cross: here, the system boundary and everything under it (read & write)

grants the attacker access to controllables observables feedback
Attacker LLM gpt-5.5 mythos-5 claude-opus-4.5 gemini-3-pro llama-4-405b

the fixed model the attacker may call

Budget $5.00 / task $1.00 $20.00 $100.00 unlimited

a per-task cost cap on LLM spend

Metrics and reports

Every run records its model, cost, and success rate to a report.

Built to scale

Run many threat models in parallel, each against its own instance of the system.

SuperRed results report for a HarmBench threat model: 42% attack success rate over 5 of 12 goals, mean score 0.57, max 0.94, $11.23 optimizer spend, run-outcome breakdown, and a table of per-task goals with outcomes, best scores, runs, and cost.

Easy to get started

Install, write a short script, run an evaluation

Point an attacker at a target, choose what counts as broken, and say what the attacker may control:

chatbot = lambda: ChatbotTarget(model="gpt-4o-mini", api_key=key)

controller = Controller(
    optimizer_factory=lambda: PAIROptimizer(),           # the attacker
    target_factory=TargetFactory(create=chatbot),        # the system under test
    security_claim=harmbench_claim(),                    # what counts as broken
    scope=frozenset({USER_TAG}),                         # the attacker's capabilities
    llm_config=LLMConfig(model="gpt-4o", api_key=key),   # the attacker's model
    task_cost_cap_usd=5.0,                               # per-task attacker budget (USD)
)
await controller.run()

The full walk-through, in the guide →

Documentation

Two ways in

Guide

Hands-on and example-driven: getting started, the core concepts of models and events, and worked examples. Start here.

Open the guide →

Reference

The framework interfaces in technical detail: the controller, optimizer, target, task, security claim, and core types.

Open the reference →

Used by researchers from