Rohan Paul· @rohanpaul_ai · X·· 9 天前AI 评分53
AI 导读
一篇关于 Kepler 的 ARC-AGI-3 研究显示，某 AI 智能体通过读取游戏的 2,172 行源码拿到 100 分，干净重跑后仅得 46.91 分。论文提出将假设表示为可执行世界模型，并通过回顾性转移检查和条件预测检查验证；作者据此建议评测智能体时设置真实访问限制并审查动作日志。
正文
This paper shows that AI agents can hit perfect benchmark scores for the wrong reasons, so audit what they did, not just the result.
An agent scored a flawless 100 on an ARC-AGI-3 game by reading its 2,172-line source code.
logs caught what the scores hid.
And then a clean rerun of that game scored only 46.91.
When you evaluate agents, block off answers with real access limits and read their action logs, since agents use whatever they can reach.
来源：Rohan Paul · x.com