Run any attacker against any AI system with any threat model
Why SuperRed
Module · Attacker
Optimizer
Drives the attack.
Framework
Controller
Orchestrates the run and enforces the security scope.
Module · System
Target
The AI system under test.
Module · Benchmark
Security claim
Defines the goal, scores the run.
Attacks and benchmarks live in one-off codebases, each wired to the system it was first demonstrated on. SuperRed pulls the pieces apart: the optimizer, target, and security claim are interchangeable modules that meet only at the controller. Swap any one and the comparison becomes a change of a single argument, not a re-implementation project.
Showcase
Over a dozen faithfully implemented attacks, including AutoDAN-Turbo, PAIR, TAP, Crescendo, and AgentVigil, alongside targets and safety benchmarks such as HarmBench, DecodingTrust, and AgentDojo.
Browse the modules →From new attack strategies to full system evaluations, SuperRed is the substrate researchers and builders use to do serious red-teaming work.
See the research →A meeting point
Attack, system, and benchmark are the same kind of thing here: interchangeable modules. Hold any one fixed to evaluate it; every contribution strengthens a shared toolbox.
An interactive environment for developing attacks, so you can build and score novel strategies on real environments as you go.
Declare your system's trust boundaries once, then see exactly which attacks succeed at each level of access.
Package a benchmark once as a security claim, then score any attacker against any target, or reuse the ones already ported.
High-level features
Switch on exactly the capabilities the attacker gets, from a single injected prompt up to full compromise, and change them between runs.
A fixed model and a per-task cost budget bound every attacker equally.
system ├─ system_prompt ├─ model_responses └─ architecture └─ high_level_architecture
user …
the trust boundaries the attacker may cross: here, the system boundary and everything under it (read & write)
the fixed model the attacker may call
a per-task cost cap on LLM spend
Every run records its model, cost, and success rate to a report.
Run many threat models in parallel, each against its own instance of the system.
Easy to get started
Point an attacker at a target, choose what counts as broken, and say what the attacker may control:
chatbot = lambda: ChatbotTarget(model="gpt-4o-mini", api_key=key)
controller = Controller(
optimizer_factory=lambda: PAIROptimizer(), # the attacker
target_factory=TargetFactory(create=chatbot), # the system under test
security_claim=harmbench_claim(), # what counts as broken
scope=frozenset({USER_TAG}), # the attacker's capabilities
llm_config=LLMConfig(model="gpt-4o", api_key=key), # the attacker's model
task_cost_cap_usd=5.0, # per-task attacker budget (USD)
)
await controller.run()Documentation
Used by researchers from
