elvis· @omarsar0 · X·· 10 天前AI 评分62
AI 导读
一篇 Meta Superintelligence Labs 论文指出，预训练 LLM 配合轻量推理框架后可作为有能力的 AI 智能体，并在测试时采样预算充足时于 pass@K 覆盖范围上超过 RL 后训练版本。后训练模型在 pass@1 更强，但会把任务推向总是解决或从不解决，论文将这种测试时可扩展性损失称为 Sharpening Tax；该现象在 BFCL v4 multi-turn、ACEBench 和 WebShop 等评测中出现。论文还提出 PTGS，在 RL 中按提示词估计难度调整采样温度，以降低该代价并提升 pass@1，论文链接为 https://academy.dair.ai/papers/sharpening-tax-in-post-training-2610.01509。
正文
Great paper for all you pre-training and post-training nerds. The surprising finding is that pre-trained LLMs, equipped with a light inference harness, can serve as capable agents and can even surpass post-trained counterparts with a sufficient test-time budget.
Banger paper from Meta Superintelligence Labs. They find something super interesting and unexpected. (bookmark it) Base models with a light harness often solve more agentic tasks than their RL post-trained versions when both get enough samples. Post-trained models win on pass@1. At large K, base models frequently solve tasks the post-trained ones never solve on BFCL v4 multi-turn, ACEBench, and WebShop. This is because post-training pushes each task toward always solved or never solved. Consistency goes up, and coverage goes down. The authors call the lost test-time scalability the Sharpening Tax. Across 42 base and post-trained pairs, it shows up in most settings, grows with model size, and can be estimated from a few rollouts. Their fix, PTGS, sets the sampling temperature per prompt from its estimated difficulty during RL. It pays a smaller tax and also raises pass@1. Paper: https://academy.dair.ai/papers/sharpening-tax-in-post-training-2610.01509在 X 查看被引用的帖子
来源：elvis · x.com