Level 5 · Flagship researchFlagship
Track A · RL + performance
Can you build a kernel-writing RL environment whose rewards hold up under optimisation pressure, and does RL on it produce real speedups on held-out operations?
Build a kernel-writing environment and grader, red-team it with frontier coding agents, benchmark frontier models on it, then RL-train an open ~7B coder and report real speedups and the hacks that emerged.
Anthropic is hiring a Research Engineer, Performance RL to advance models’ ability to write correct, fast accelerator code by building RL environments and evaluations – this project is that job. The take-home README warns that models cheated by editing tests or re-enabling disabled settings; hack-proof grading is the hard, unsolved part.
01Read the job and the cheating warningsM1 · week 8
Read the Performance RL posting and the take-home README’s notes on how models cheated. Write down every way a grader could be fooled.
02Correctness gradingM1 · week 8
Check correctness on randomised shapes and inputs with numerics tolerances, so fixed test cases can’t be memorised.
03Isolated timing and tamper detectionM1 · week 8
Time kernels in isolation with enough repeats to beat noise, and detect tampering with the harness, settings or tests.
04Red-team it with frontier coding agentsM1 · week 8
Let agents try to game the grader. Publish every exploit found and how you closed it.
05Benchmark frontier models and publishM1 · week 8
Run frontier models on the environment, publish a leaderboard and the exploit catalogue as the week-8 milestone.
06RL-train an open ~7B coder
Train on the environment with your hardened grader.
07Report real speedups
Measure against torch eager and torch.compile on held-out operations, not the training ops.
08Document training-time hacks
Which hacks emerged during training, and which grader fixes closed them.
09Release
Open environment and leaderboard, exploit catalogue, RL results.