跳到正文
Rohan Paul· @rohanpaul_ai · X·· 3 小时前AI 评分46
AI 导读

研究发现,廉价验证器 Jev 在检查 RAG 声明时准确率可接近前沿 LLM,但会放过更多不支持的声明。在 495 个标注声明的测试中,Jev 的准确率仅比 GPT-6 Astra 低 0.6 个百分点,成本降低 187 倍,但其批准的不支持声明比例达 23.2%(Astra 为 14.1%)。开发团队建议在 RAG 产品中选择声明检查器时,必须测量其批准不支持声明的频率。

正文 · 原文

This paper finds that a cheap verifier can match a frontier LLM's accuracy at checking RAG claims, while letting more unsupported claims slip through.

RAG pipelines keep asking whether evidence supports a claim, a simple yes or no. Most teams get that answer from a text-generating LLM at generation prices.

They tested Jev, which returns only a label and a probability, against GPT-6 Astra on 495 human-labelled claims. They also built a cascade where Jev answers confident claims and passes the rest to Astra.

Jev landed within 0.6 points of Astra's accuracy at 187 times lower cost, but it approved 23.2% of unsupported claims against Astra's 14.1%. The cascade also cut Astra's bill, yet sending the same share of claims to Astra at random saved about as much.

Before picking a claim checker for a RAG product, measure how often it approves unsupported claims, since headline accuracy hides that.

来源:Rohan Paul · x.com