Daily AI Paper

Training agents with reinforcement learning is hard when the only feedback is a single reward at the end of a long task. This paper tackles that problem for multi-turn software and web agents by using a teacher-student setup: a skill-conditioned teacher first learns strong behavior, then a student learns from both reinforcement learning and dense token-level guidance. The key twist is that the teacher does not stay in the loop forever. The authors introduce self-retiring on-policy distillation, where the student watches the teacher only until its own behavior has caught up enough, then automatically drops the distillation signal and continues with reinforcement learning alone. That matters because a teacher can be helpful early on, but later it may become a bottleneck or even mislead the student. Across several model sizes, this approach substantially improves success on ALFWorld and WebShop, and in some settings the student even surpasses its teacher. It is a practical step toward training agents that learn faster, use supervision more efficiently, and keep improving after the easy gains are gone.