Level 5 · Flagship researchFlagship
Track D · RL infrastructure
When agent trajectories run for minutes with sandboxed tool calls, how should asynchronous RL handle policy staleness inside a trajectory, and which corrections keep training stable?
Add a multi-turn, sandboxed agentic workload to an open async RL framework (AReaL or verl), measure throughput against a synchronous baseline, then run a systematic staleness study and get it merged upstream.
Fully asynchronous systems report large speedups – AReaL up to 2.77×, DORA 2.12× end-to-end – and DORA names intra-trajectory policy consistency, data integrity and bounded staleness as conditions for convergence. Most of this work is on maths reasoning; Anthropic’s RL Research Engineer postings ask for RL infrastructure, sandboxed code execution and distributed systems.
01Read AReaL, DORA and AsyncOPDM1 · week 8
Note each system’s staleness handling and convergence conditions.
02Add a multi-turn sandboxed agentic workload to AReaL or verlM1 · week 8
Keep one research question at the centre – getting lost in infrastructure is the main risk.
03Measure throughput and GPU utilisation, then publishM1 · week 8
Against a synchronous baseline on the same hardware. This is your week-8 milestone.
04Systematic staleness study
Mid-trajectory weight updates and partial rollouts, across seeds.
05Failure catalogue
Entropy collapse, NaNs and other failures, each with a diagnosis.
06Get it merged upstream
Turn the workload and fixes into PRs the maintainers accept.
07Release
Merged PRs, benchmark report and stability study.