← Migration

Migration to v0.3.0

This release reworks what the Controller does with its output: it now streams live progress while a run is in flight, and it persists a structured, resumable result tree by default (persistence used to be opt-in). Nothing about the attack loop, the scope model, or the event/trajectory types changed.

Live progress reporting replaces the end-of-run summary

The Controller no longer prints a single summary block at the end via a private _print_summary. It now streams live progress throughout the run, controlled by report: a colored live dashboard on an interactive terminal, plain lines otherwise.

One new constructor argument drives it:

• report: bool | Literal["auto"] = "auto". True/"auto" show progress; False is silent. On a real interactive terminal, progress is a colored live dashboard; on a non-TTY, CI (CI set), NO_COLOR, a dumb terminal, or piped output it degrades automatically to plain lines. The plain start banner and end summary reproduce the content of the old _print_summary, so nothing is lost when the canvas is unavailable.

Multiple Controllers run concurrently through asyncio.gather share one dashboard, one row each. rich (>=14,<15) is now a core dependency, pulled in automatically by pip.

Migration: pass report=False to restore silence. Nothing else is required; existing calls keep working and simply gain live output.

Persistence is ON by default; new persist flag to turn it off

Persistence used to require passing results_dir; the default wrote nothing. It is now on by default and writes to a results folder.

• persist: bool = True. Set persist=False to write nothing (the old default behavior).
• results_dir is now the results root (the parent that holds one folder per experiment), not a single file’s directory. When omitted it resolves to the SUPERRED_RESULTS_DIR environment variable, else ./superred-results/.
• overwrite: bool = False forces a full recompute of an existing experiment (see resume below).
• attacker_label / target_label / claim_label (all str | None = None): short human names used in the experiment folder name and the dashboard. Each falls back to a factory/class name, else a generic default.

Migration: to keep the pre-0.3.0 “writes nothing unless asked” behavior, pass persist=False. Add ./superred-results/ (or your SUPERRED_RESULTS_DIR) to .gitignore.

v4 directory layout replaces the flat v3 {scope}__{model}.json files

SCHEMA_VERSION is bumped 3 to 4. The old two-file-per-threat-model layout (a flat {scope}__{model}.json summary plus a {scope}__{model}/NNNNN__goal.json detail folder) is replaced by one self-describing directory tree per experiment:

{results_folder}/
├── experiments.json                     # cross-experiment index (sweep landing)
└── {slug}-{hash8}/                       # one experiment (one threat model)
    ├── manifest.json                     # index: params + summary + tasks[]
    ├── result.json                       # claim-level final metrics (completion marker)
    ├── logs/diagnostics.log
    ├── tasks/
    │   └── 00001__{goalslug}/            # latest result for this task
    │       ├── task.json                 # per-task result + metrics
    │       ├── iterations.json           # per-run score/metric progression
    │       ├── trajectories/run_00001.json
    │       └── logs/diagnostics.log
    └── previous_01/                       # immutable snapshot of a prior run

{slug} is attacker__target__claim__model (short, human, and does not list scope tags); {hash8} is 8 hex of a sha256 over the measurement identity (attacker/target/claim/model, scope, read_only, budget, max_runs, feedback). The identity excludes the schema version, so a framework upgrade still resolves to the same folder and resumes.

Migration: any code that read the old {scope}__{model}.json files must move to the reader API below; the on-disk shape and filenames have changed entirely.

New public reader API in superred.core.persistence

Reading results is no longer “open the JSON file you named”. The persistence module now exposes public readers for analysis code (and the results website):

• load_experiments_index(results_root) (the cross-experiment index),
• load_manifest(experiment_dir), load_result(experiment_dir),
• iter_tasks(experiment_dir) (scalar TaskViews, one per task),
• load_task(task_dir), load_iterations(task_dir), load_trajectory(task_dir, run_number).

Migration: replace hand-rolled JSON reads with these functions.

Resume: re-running the same experiment keeps completed tasks

Re-running the same Controller (same measurement identity, hence the same {slug}-{hash8} folder) resumes instead of colliding. Tasks whose prior result was a valid measurement (success, failed, or budget_exhausted) are kept and not re-run; only error, interrupted, or missing tasks recompute. A task is matched to its prior result by (same index + same goal content hash), so appending tasks to a claim resumes the existing ones. overwrite=True forces a full rerun.

This replaces the v3 behavior, where a pre-existing summary file or task subfolder raised FileExistsError. Reruns are crash-safe: before any change, the prior complete state is snapshotted immutably into the next previous_NN/ (kept tasks are shared in by hardlink), so an interrupted rerun never destroys the only copy.

New metrics fields on results and in persisted JSON

• RunResult.run_usage_delta: LLMUsage: this run’s own LLM usage (the cumulative llm_usage minus the previous run’s snapshot). Summing deltas across a task’s runs equals the task total; summing the cumulative llm_usage over-counts. Use run_usage_delta for per-run cost; keep llm_usage for budget-versus-performance curves.
• Timestamps: started_at / ended_at (datetime | None) on RunResult, TaskResult, and ThreatModelResult.
• Per-run flags on RunResult: evaluated (score came from the evaluator, vs. a synthetic zero on an error/budget path), errored, done.
• Summary: the persisted summary (and the dashboard) now carry an explicit attack-success rate and a stop-reason histogram: asr, n_completed (= done + max_runs + budget_exhausted), n_success, n_failed, n_error, n_budget_exhausted, n_skipped, max_primary_score, mean_primary_score, total_llm_usage. asr = n_success / n_completed, so errored and skipped tasks are excluded from the denominator.

These are additive on the result dataclasses (all defaulted), so reading existing fields is unaffected.

Persisted content is sensitive (unchanged risk, now default)

This is a red-teaming framework, and persistence is now on by default, so this caveat matters more: persisted trajectories contain jailbreaks, planted secrets, and exfiltrated content, and are NOT scrubbed. LLMConfig still serializes {model} only (api_key/api_base are never written), but treat the whole results folder as sensitive.

A generic results website ships with the framework

The framework ships a single self-contained static HTML dashboard for the results tree. It reads the JSON files, shows the metrics, filters tasks by outcome (success / failure / error), and drills into each task’s runs and trajectories. It is generic and needs no build step.