Perplexity 与英伟达联合推出了 JOLT 自蒸馏与强化学习训练法，通过对教师模型施加惩罚以防止其偏离学生模型过远。实验表明，JOLT 在 MATH 评测基准上以少 6.5 倍的训练 token 数达到与 GRPO 相当的表现，而 JOLT+ 则在数学、编码、终端任务和工具调用评测基准上显著超越 GRPO。
Sharing new post-training research on self-distillation + RL (JOLT), done together with @NVIDIAAI.
In self-distillation, the same model performs the role of both a teacher and a student. The teacher gets extra information that the student doesn't, and the student learns from feedback provided by the teacher. However, the teacher could still use shortcuts in ways the student has no way to follow.
Our proposed approach, JOLT, trains the teacher as well, to learn from rewards on its own attempts, while penalizing any drift that's far off from what the student model would do. JOLT+ further supplements JOLT by providing the student model outcome-based rewards in addition to the teacher supervision signals.
On the MATH benchmark, we found that JOLT matches GRPO with 6.5x fewer training tokens, while JOLT+ convincingly beats GRPO across math, coding, terminal tasks, and tool use benchmarks.
来源：Aravind Srinivas · x.com