Level 5 · Flagship researchFlagship
Track B · Alignment auditing
Can you build an auditing method that finds hidden behaviours in the models that current auditing agents miss?
Reproduce the AuditBench agent baseline on its hardest slice, analyse why each audit fails, then build a new auditing technique validated on held-out model organisms you train yourself.
AuditBench (Anthropic Fellows + Anthropic, 2026) released its models, auditing agent and evaluation framework. Black-box tools worked best; demonstration-trained models were harder to audit than synthetic-document ones, and adversarial training made them harder still. Anthropic’s April 2026 risk update reports a model organism that evaded its alignment audit.
01Set up AuditBenchM1 · week 8
Get the released models, auditing agent and evaluation framework running.
02Reproduce the baseline on the hardest sliceM1 · week 8
Demonstration-trained and adversarially trained models.
03Analyse every failure and publishM1 · week 8
A detailed analysis of why each audit failed – this is the week-8 milestone.
04Design a new technique
White-box or black-box, aimed at the failures you found.
05Train held-out organisms
Train your own model organisms, including new evasion strategies, that the technique never saw during development.
06Validate and report where it still fails
Beat the baseline on the hardest slice and on your held-out organisms; be explicit about failures.
07Release
Method and code, your new organisms, and results on the held-out set.