NVIDIA 论文提出通过首个通过测试的编辑评估 base 模型以预测智能体后训练表现
NVIDIA 论文提出在首个让测试通过的编辑处评估 base 模型，预测编码智能体后训练表现
NVIDIA 研究提出，在智能体轨迹中仅评估模型在首个使测试通过的编辑处的表现，比逐步打分更能准确预测其后训练潜力。该方法通过回放成功轨迹并检查模型写出关键编辑的概率，与模型在 SWE-bench Verified 上的后训练得分相关性高达 0.964，且仅需基准测试数据即可运行。
Rohan Paul· @rohanpaul_ai · X·2026-10-12 23:39· 9 分钟前AI 评分50
AI 导读
NVIDIA 论文提出，相比逐步打分，只在首个让测试通过的决定性编辑处评估 base 模型更能预测其智能体后训练潜力。方法回放成功智能体轨迹并在每次编辑后重跑测试，检查 base 模型是否会写出该编辑、在失败编辑中选中它并写出可通过的实现。模型写出该编辑的概率排序与十个模型后训练后的 SWE-bench Verified 得分相关性达 0.964，且只需基准的成功轨迹和测试即可运行。
正文
New Nvidia paper shows scoring base models on every step of an agent run works worse than scoring them only on the edit that first makes the tests pass.
Post-training is expensive, so picking the right base model matters. Yet agent benchmarks score base models near zero, and simple coding tests barely predict later agent skill.
They replay a strong agent's successful run and rerun the tests after each edit to find the 1st edit that makes them pass. At this decisive step they test whether a base model favors that edit, picks it over failing edits, and can write a passing one.
How likely each base model was to write that exact edit ranked the same 10 models in line with their post-trained SWE-bench Verified scores, at 0.964.
When choosing a base model to post-train for agent work, run this check first. It needs only a benchmark's successful agent runs and its tests.
来源：Rohan Paul · x.com
来源：AIHot · aihot.news