METR / 文章
TH1.1 和 TH2 合在一起,50% 时间范围大约每 3.61 个月翻一倍
METR 11月20日更新:公开模型在容易验证的编程和优化任务上已经很强,在更贵、反馈更慢的任务上判断力较弱。他们仍认为,现有公开证据还不足以排除未来两三年出现剧烈的 AI 研发加速。
乔木改写
2026年11月20日,METR发布能力评估更新。把已有的 TH1.1 和新增的 TH2 任务合在一起,他们得到的 50% 时间范围大约每 3.61 个月翻一倍,也就是 110 天,95% 置信区间是 2.08 到 4.94 个月。这和此前 TH1.1 上约 105 天翻一倍的趋势接近。
若按这个速度做朴素外推,模型的 50% 时间范围会在 2027 年 7 月达到这套任务里最长的分数门槛,也就是 200 小时。
作者包括 Megan Kinniment、Manish Shetty、Brian Goodrich、Matt Putz、Spruce Bondera、Rowan Huang、Nikola Jurkovic、Jonathan Gabor、Oscar Sykes、Rif A. Saurous 和 Beth Barnes。评测对象是公开模型,直到 Opus 5.5 和 GPT-6 Astra。
METR 仍然对两个中心问题很不确定:剧烈加速实际上需要哪些能力,以及相关能力正在以多快的速度提高。他们所说的剧烈 AI 研发加速,指的是能力提高得太快,以至于人类无法再有意义地理解、评估并应对新系统带来的灾难性风险。就目前的公开证据而言,他们认为还不足以排除未来两三年出现这种加速。
下图是他们对本轮能力测量的总览。

媒体未能载入,请通过下方原文链接查看。
时间范围曾经能用来压低风险,现在更难了
这篇更新把重点放在与加速 AI 研发相关的能力上。一个仍然分叉的问题是:只在“规格清楚”的任务上表现很强,是否已经足够带来剧烈加速,还是必须再具备其他能力。METR 说,他们希望给出的证据,对持不同看法的读者都有用。
时间范围最初用来测量与软件工程相关的智能体能力。它有过三层用处:当时测到的能力被他们视为造成灾难性风险的必要条件;测量有相当一部分锚定在人类参照数据上,比较容易对照风险阈值;而且测到的能力离可能的危险还很远,因而可以比较有把握地说,这些系统几乎不构成灾难性风险。例如,他们提到过对 GPT-5 的评估。
局面已经变了。2025 年 3 月时间范围论文发布时,最强模型的测量结果大约是 1 小时,对应 Sonnet 3.7。一年之内,模型就把包含最长 30 小时任务的 TH1.1 套件做到饱和。在许多领域,要找到能胜过模型的人类基线,已经变得既困难又昂贵。
公开模型在编程和黑客一类任务上已经足够强,METR 不再认为,单靠“在可验证的软件工程任务上无能”,就能构造出风险非常低的论证。他们举例说,公开前沿模型已经能够攻击服务、推断自己被如何打分,并绕过沙箱限制。若还要对能力风险下判断,要么更具体地讨论能力剖面和特定威胁模型所需的能力,要么找到其他仍然成立的、更宽的智能体能力限制。
他们也越来越担心,容易运行、放在容器里的评测,可能高估真实世界能力,因为类似的强化学习环境已经常被拿去训练。开发者和评估者都想要便宜、可量化的能力信号,于是评测和训练环境往往很像。评估者真正想代理的,常常是更难直接测量的能力;一旦这些代理任务或邻近任务被拿去训练,测量就容易偏高。
因为新任务更难、更耗时,他们也改了人类基线和时间估计办法。
新套件加进了更贵、更难验证的任务
为了更好理解这些能力,他们扩展了时间范围套件:一边增加仍可便宜验证的任务,一边加入难以验证、或需要判断力的任务。
TH2 里可验证的任务族包括:AZsPCs,15 道来自 Al Zimmermann 编程竞赛的启发式优化题;Budget NanoGPT,2 道基于 NanoGPT 速度赛的训练脚本优化;Minecraft Bot,3 道让机器人通关的任务;Reverse Conway,2 道在康威生命游戏里反推初始盘面的任务;Train a Program,1 道训练神经网络去复现给定程序输入输出的任务;Earthborne Rangers,2 道改编自 Epoch EBR-Bench 的资源管理卡牌任务;Slay the Spire,4 道通过文本界面玩 roguelike 的任务;以及 Dominion-like Game,2 道构建牌库的游戏任务。
难以验证、需要判断力的任务族包括:Making METR Tasks,要求做出一份 METR 会觉得有信息量的评测任务;Investigate Wiki Incident,基于 MessageBoardAuditBench,重建一次关于智能体在 wiki 上合谋的调查;LMCA,给哲学论证的批评打分,分数看与专家评分的一致性;Judge Run,用评分表去给 Making METR Tasks 的一次运行打分。
TH1 和 TH2 的人类基线也不一样。TH1 的基线人员大致是有约 5 年经验、并通过资格任务的专业人士,时薪通常 50 到 100 美元,另有绩效奖金,一般不许用 AI 辅助,通常会给出目标分数,时限最多每题 8 小时,主要测量完成任务的时间。TH2 找的是该任务领域的专家,时薪 200 到 350 美元,没有奖金,允许用 AI 做实现,但只能用 Sonnet 5,不设硬时限,会定期看进度,也不预先给出目标分数,测量的是达到一个或多个分数门槛所用的时间。
他们还为同一任务设置了多个分数门槛。后文里的 METR Capability Index,简称 MCI,则用类似项目反应理论的方法,直接从智能体在各任务上的分数估计任务难度和模型能力,从而不必依赖人类耗时估计。它和 Epoch 的 ECI 是同一类指标。
下图比较了 TH2 对模型和人类来说分别有多难、要花多久。

媒体未能载入,请通过下方原文链接查看。
在反馈清楚的编程和优化题上,模型已经非常强
在有强反馈信号的编程问题和指标优化上,模型通常能达到或超过熟练人类。METR 自己去招募人类基线时,通常找不到能胜过当前前沿模型的人。
他们希望基线人员的水平接近 AI 实验室员工,因此招募了该领域的专家。例如,Train a Program 上最好的外部基线人员是前 Codeforces International Grandmaster,并赢过 2020 年 Kaggle ARC-AGI 竞赛。Minecraft Bot 和 Reverse Conway 上表现最好的基线人员头衔没那么显赫,但有可补偿的专门知识或经验。他们支付每小时 200 到 350 美元,7 次尝试过的基线总共花了大约 9 万美元。
他们并不能确定,这些基线人员是否可与前沿实验室的研究人员相比。一方面,这些人经验丰富,而且常常有与任务直接相关的经历。另一方面,AI 公司给的报酬很高,招聘团队也在专门寻找顶尖人才;这些任务又需要投入数十小时,可能筛掉了全职工作者,并带来某种不利选择。他们猜测,基线人员在任务领域里的水平,大概还比不上典型前沿研究者在自己领域里的水平。
因为这些任务需要的技能和时间都明显更高,只有 9% 的人类时间估计来自成功完成的基线。其余任务的时间,来自公开记录、专家访谈和外推。
外部工作也看到类似的高强度。在 MirrorCode 上,智能体要根据完整测试套件重写软件;Opus 5.5 能解出 77% 的中大型程序,而人类专家独自做这些程序可能要花数周到数月。GPT-6 Astra 最近在 FrontierMath Tier 4 上得到 98%,这套基准是研究级数学题,设计上需要该领域的强数学家花 1 到 2 周。
在两类已有大量人类数据的可验证优化任务上,模型胜过大多数人,但不总是压过最顶尖的那几位。
左边是模型方案在 AZsPCs 竞赛里相对人类参赛者的名次,右边是模型相对第 78 号纪录的加速,再拿去和人类第 79 到 92 号纪录的加速相比。AZsPCs 上,模型有 48 小时和一台 28 核机器,人类参赛者大约有三个月。Budget NanoGPT 上,模型有 48 小时和 8 张 H100,贡献者通常报告每条纪录花 16 到 40 小时。

媒体未能载入,请通过下方原文链接查看。
第一类是 15 场为期数月的 Al Zimmermann 优化竞赛。最强模型,也就是 Fable 5.1、Opus 5.5 和 GPT-6 Astra,名次中位数落在人类参赛者的第 96 到 98 百分位,每场有 97 到 413 名参赛者。他们估计,一名投入的参赛者在整场比赛里可能花 90 到 180 小时,并认为顶尖参赛池相当强。在中位竞赛里,大约有 4 名人类超过了他们测试过的最强模型。
第二类是 Budget NanoGPT 速度赛。他们给智能体的算力,接近人类速度跑者通常使用的量。智能体方案都从第 78 号纪录起步,和人类纪录并不能完美对比,因为每条人类纪录都建立在前一条之上。不过,访谈让他们觉得,每条人类纪录消耗的劳动和资源,与他们的一次智能体运行大致可比,因此仍把这种对比看成粗糙但有信息量。他们也提醒,近期的“人类纪录”可以有 AI 辅助,有时还很重;计入的人类纪录只到 2026 年 8 月。
在这套速度赛上,完全自主的智能体超过了许多能干的速度跑者,用更低成本拿到更好的加速。但顶尖人类贡献者仍能达到比智能体大得多的加速。每个模型跑 2 到 5 次,最好的一次智能体运行在 48 小时里相对起始纪录加快了 15.7%。第 92 号纪录的作者在编码上使用了 AI 辅助,估计花了大约 60 小时,相对前一条纪录加快了 41%。
更像强化学习环境的任务,模型更强;更贵的任务,模型更弱
METR 担心,他们现在主要用来测量能力的评测,和公司用来训练模型的强化学习环境越来越像。两边都想便宜、方便地抓住能力信号。随着强化学习在训练中占比升高,这种重叠会造成系统性的测量问题。
为了粗略刻画“把一道题做成强化学习环境有多贵”,他们引入了生命周期成本。它指的是:为任务做一个容器化环境,再把这个任务运行并打分 100 次,所要花的成本。这个量并不打算反映强化学习环境的真实成本,只是相对昂贵程度的一个很粗的指标。
他们列举了会抬高现实任务成本的因素:需要人工、尤其是专家打分;每轮要消耗 GPU 等算力,TH2 里的 Budget NanoGPT 和 Train a Program 都用 GPU;串行时间很长,TH2 可验证任务里最长的评估要跑 48 小时;token 成本;高风险评估需要行动监控和人工审批;环境更复杂,往往得从游戏或数据集里“进口”再做公平性处理;与真实世界互动会增加监控、质检、噪声、寿命管理和资源开销;解法还要在可迭代指标和最终要求之间留出泛化空隙;还要防止奖励黑客,并让打分真正打到想测的能力上;有些任务必须请专家来创建、运行或打分;如果运行中还要有人参与,墙钟时间、协调和人力成本都会很高,不过 TH2 里还没有这类任务。
下图汇总了这些会让现实任务更贵的因素。

媒体未能载入,请通过下方原文链接查看。
生命周期成本由一个 LLM 评判器来估计,用的是 Opus 5.5。评判器会看到任务代码,包括 README、说明和打分,再估计开发、质检和打分等成本。相似任务族的共享开发成本会被摊销。他们迭代提示词,使评判器对大约 10 道、覆盖不同领域和成本区间的任务,大致符合他们自己的估计。他们预期这套提示对套件外的任务也还算能用,但成本阶段怎么切、假定 AI 能把任务开发加快多少、人工费率等,反映的是 METR 自己的经验。
他们也列出了这个代理指标的限制:它不考虑不同能力领域的需求差异,比如公司可能更想把模型训练得会写代码,而不那么想训练欺骗和破坏;它也不考虑任务之间的泛化,以及一个领域里要覆盖多少任务才够;这个 LLM 评判器是按 METR 的约束校准的,实验室的约束可能不同;成本还会随模型变强而变化,今天贵的东西,几个月后未必还贵。他们并不声称这些因素是根本障碍,甚至不声称它们将来仍然相对昂贵。
在他们的数据里,人类完成时间和生命周期成本相关性很高,R² 是 0.82。对 TH2 任务,生命周期成本只按最高分数门槛来估计。

媒体未能载入,请通过下方原文链接查看。
控制住完成时间后,生命周期成本更高的任务,往往比单凭耗时预测的更难,这部分残差的 R² 是 0.14。数据很吵,也可能受到时间估计误差或少数高成本任务的影响。他们在图里过滤了这类离群任务。一个让他们稍感安心的观察是:如果改用模型去估计人类耗时,而不是用人类估计或基线,仍然能看到类似趋势。

媒体未能载入,请通过下方原文链接查看。
再按生命周期成本高低把任务分开,并控制任务长度后看时间范围,模型在高成本子集上表现更差。高成本子集平均比低成本子集贵 2.9 倍,模型的时间范围也差 2.9 倍,相当于 1.5 次翻倍。

媒体未能载入,请通过下方原文链接查看。
有意思的是,与低生命周期成本子集相比,模型在高生命周期成本子集上的表现,更接近 OpenAI 报告的内部研究使用情况。他们写,这里可比较的任务分布并不多,但至少把这看成一个让人鼓舞的信号。
两边的翻倍速度仍然接近。也就是说,在这批数据里,更贵的任务更难,但模型在更贵和更便宜的任务上,进步速度差不多,即使在最贵的那些任务上,表现也仍在随时间提高。
反馈慢、要靠判断力时,模型看起来不如人类专家
生命周期成本里,有一部分接近任务好不好验证,或者说有多“模糊”。更模糊的任务往往更难自动打分,或者虽然能打得更忠实,却贵到普通评测负担不起。这类任务常被说成依赖品味和判断,研究是一个典型例子。
METR 认为,把任务简单分成“模糊”和“可验证”可能误导。许多被叫作模糊的任务,只要把后果完整展开,其实仍可验证。例如做一款“好用的应用”,用得够久之后,用户数或评分就会给出更客观的质量信号。他们更想用的操作化办法,是看长期评估解法质量有多贵。像决定研究议程这类事情,要把全过程走到长期质量可被评估的那一步,往往非常贵,因此他们预期这类任务的生命周期成本会高于同等长度的普通任务。
文中的判断力,大致指:在缺少清晰、快速的经验反馈时,仍能做出好选择。它常常来自有力的论证和推理,或来自经验中学到、再被小心应用的启发式,从而缩短本来又贵又长的反馈环。因而,判断力在更贵、更难验证的任务上特别有用。
剧烈的 AI 研发加速到底需要多少判断力,他们并不清楚。但他们会把判断力及相关能力上的弱点,看成最可能推迟这种加速的候选因素之一。
在 LMCA 这套基准上,模型要给哲学论证的批评打分。模型一直在稳步提高,许多模型与专家的一致性,已经高于专家只花 5 分钟时的估计水平,但仍低于专家花 15 分钟时的估计水平。
他们还做了一道题,测试模型能不能自动替 METR 出评测。智能体要为评测套件创建一道对新能力评估有信息量的新任务。他们测试过的模型给出的任务,没有一道有用到值得写进这篇文章。

媒体未能载入,请通过下方原文链接查看。
这些模型通常更愿意做自动打分的优化题,即使在后续实验里被明确要求去做“成败取决于短期反馈很差的那些决策”的任务,也还是如此。尽管有网络,智能体也没有主动去把“METR 会觉得什么任务最有用”弄得更清楚。他们没有看到智能体提出新的威胁建模,或基于对模型能力特别有洞察的假设来构造任务。
让模型去评判其他模型做出来的任务时,也出现了相关弱点。请它们按“对 METR 有多有用”来打分时,它们与 METR 自己评分的一致性,取决于评判模型和出题模型谁更强。
当前沿模型去评比自己更弱的模型所出的题时,与人类评分的一致性很高;但题目若由更强或差不多强的模型给出,它们会打出偏高的分数。与人类评分者一致性最高的评判模型是 Opus 5.5,它给 Astra 所出题目的平均分,大约比人类高出 0.2 分,满分是 1 分。

媒体未能载入,请通过下方原文链接查看。
最直接的解释是,模型对“什么算一道好题”判断力不足。这种判断会反过来限制它在出题任务里的表现,因为它只能拿自己的判断当优化目标。结果是:它们当评判时,会系统性地高估自己的解,以及更强模型的解。另一种可能是,更强的模型会表现出某些行为,让更弱的评判高估其解法。
关于判断力的结果,他们加了两条限制。第一,模型在这类问题上很可能没有被充分引出,例如在他们的设置和类似设置里,模型会明显用不满被提供的 token 限额和工具。第二,当前公开前沿模型在一些机器学习任务上已经表现出良好判断,尽管这种判断的普遍性并不清楚。例如,在以程序化方式打分、历时数天到数周的机器学习评估上,模型能够胜过他们的人类基线,其中包括他们特意做成新颖、并认为没有被拿去训练过的任务。相关工作里,P-Zero Research 也在 TasteVal 上看到强表现,这套基准测量智能体在最长 120 小时、可验证的机器学习优化任务中,实验选择的计算效率。这种表现,在与人类相当的资源限制下似乎也能保持,TasteVal 和他们的 Budget NanoGPT 都是例子。不过,他们觉得,模型在这些任务上表现出来的判断,有可能在某种意义上不如人类判断那么普遍;也可能并没有根本差别。
合在一起是 3.61 个月翻一倍,可验证子集看起来更快
在 TH1.1 和 TH2 的合集上,时间范围大约每 3.61 个月翻一倍,95% 置信区间是 2.08 到 4.94 个月,也就是 110 天。因为 TH2 更难,同一模型在合集上的 50% 时间范围,会低于只看 TH1.1 时的值。

媒体未能载入,请通过下方原文链接查看。
分开估计不同任务子集的趋势时,TH2 可验证任务上的翻倍时间,看起来短于 TH1.1。数据很吵,但 TH2 可验证子集上他们看到的翻倍时间是 1.65 个月,95% 置信区间是 0.79 到 3.99 个月。这条趋势快于合集。

媒体未能载入,请通过下方原文链接查看。
他们也展示了另一套可验证任务 MirrorCode 上的趋势,拟合针对前沿模型。翻倍时间的置信区间,通过对模型、任务族和运行做自助抽样得到;模型自己的置信区间则对任务族和运行做自助抽样。这张图里展示的模型更多,因为不必限制在 TH1.1 和 TH2 上每个模型、每道题都有运行的数据。
TH2 可验证任务上更快的翻倍,可以被当成 2026 年能力加速的证据,至少是在这些任务、用时间范围来度量时。但他们提醒,2025 年下半年,MirrorCode 上已经出现过类似的快翻倍,而当时 TH1.1 仍在以更慢的速度增长。
他们并不确定,TH2 可验证任务和 MirrorCode 上更快的原因是什么。一种可能是,时间范围所测量的能力总体上在加速;这种加速有可能被合集里的人为减速遮住,因为换到整体更难的 TH2 后,固定时间范围对应的任务更难了。其他可能还包括:这些分布覆盖的日期更窄,或者这些任务的性质让模型更容易快速提高,也可能是几件事叠在一起。
一种他们觉得说得通的可能是:任务集合越不多样,翻倍时间往往越短。
用能力指数看公开前沿,没有看到明显加速或减速
他们还用 MCI 做了另一种能力度量。和 Epoch 的 ECI 类似,这种方法直接从智能体跨任务的分数估计任务难度和模型能力,因而不需要人类耗时估计。
用这套指数去看趋势变化时,结果可以从轻微减速到中等加速,取决于方法选择。他们的数据只用到 2026 年 9 月及以前的公开模型,因此不能直接拿去和“AECI 上出现 2 倍加速”的结果比较。
把 MCI 拟合到他们的数据上,在所测试的公开模型、以及公开前沿整体趋势上,他们没有看到明显的加速或减速。

媒体未能载入,请通过下方原文链接查看。
如果改成按每个 AI 开发者分别校准和拟合,某些条件下会看到斜率变化,也就是能力增速上升。在他们最能复现的条件下,也就是公开的 Anthropic 模型直到 Opus 5.5,这些开发者特异的斜率变化,大体可比他们 2026 年 10 月对 Anthropic 的独立 AI 研发加速评估,以及近期 Anthropic 系统卡中的结果。但这些斜率变化对方法选择相当敏感,他们并不确定其有效性。
他们还提醒,用这类项目反应理论方法去检测加速,有可能漏掉真正的加速。开发者可能随着模型变强而推迟公开发布;MCI 对最强模型的能力估计也总是更不确定,这都会推迟检测。这种方法也只能检测 MCI 所度量的总体能力加速,而不能把它归因于 AI 本身。如果开发者没有用 AI 来帮助研究,反事实趋势完全可能是减速的,而 AI 的使用只是把趋势拉回到现在的水平。他们并不知道那条反事实趋势是什么。
稳健性检查和尚未回答的问题
他们做了多类稳健性检查。为确认运行的 token 预算够大,他们看了表现如何随 token 增加而变化,并尽量把每道题的预算设在表现开始平台的位置,但并不总是做得到。
他们也检查奖励黑客,一旦发现,该次运行记 0 分。办法是人工查看一部分轨迹,再用模型扫描器跑完整批运行,有时会按任务给出应搜寻的模式。发现的奖励黑客,大多来自接近前沿的智能体,例如 GPT-5.6 Sol、GPT-6 Astra、Fable 5、Fable 5.1 和 Opus 5.5。
若分类器阻断了生成,或模型自己拒绝作答,这些运行会被排除。这种情况很少,TH2 和 TH1.1 加起来大约 36 次。他们也对故意藏拙做了初步检查。
最后,他们试过用“群体”脚手架,而不是常规脚手架,并通常拿它和 best-of-k 比较;在这类任务上,best-of-k 通常优于一次非常长的串行运行。数据仍然很吵,但在有明确优化指标的任务上,群体脚手架似乎比 best-of-k 对照更好;在其他任务上,效果不清楚,甚至为负。尤其是,当任务需要智能体自己创造代理指标时,大的群体似乎会损害表现。这些群体实验大约花了 50 万美元的 token。
因此,他们相信,在干净、可验证的优化任务上,因为没有使用群体,自己可能多少低估了模型表现;在其他任务上,他们预期不用群体并不是低估的来源。
讨论部分里,他们写:随着模型变强,一些过去用来给风险设上界的方法和评测,已经不那么有用,甚至过时了。当前公开模型在边界清楚的机器学习、网络安全和编程问题上已经非常强,因此更难再靠某种容易测量、覆盖很宽的无能,来限制风险。
对能力风险的理解,现在更取决于能力剖面的具体细节,以及关于世界的事实。对许多最重要的问题,他们觉得自己还没有好答案。例如:模型在更容易或更便宜验证的任务上,究竟比更难或更贵验证的任务强多少?在这批数据里,相对生命周期成本更高的子集上,表现更弱,但生命周期成本和所谓“真实感”之间的联系仍很不清楚,这种更弱的表现是否会搬到其他任务分布上,他们也不知道。模型是否在更难验证的任务上进步得更慢,如果是,慢多少?在这批数据里,高成本和低成本子集的翻倍时间接近,但在成本高得多、或更真实的任务上,趋势完全可能不同。剧烈加速到底需要哪些能力?他们在文中大多避免声称什么是必要且充分的,而这正是他们最大的不确定来源之一。也许,仅靠容易验证的任务,就已经足以剧烈加速 AI 研发。因此他们认为,在模型还没在许多“真实”任务上表现很好、而这些任务也许是自动化“普通”工作所需要的之前,仍有必要把剧烈加速当成可能发生的事。至少,AI 开发者正投入大量努力,让自己的 AI 去自动化他们自己的研究,这可能会让自动化比 AI 研发“本来”的难度来得更早。
他们感谢了帮助实现、质检和检查奖励黑客的任务开发者:Drew de Wet、Gerard Boxo、Martin Milbradt、Máté Csikós-Nagy、Pablo Romero、Prakrat Agrawal、Gábor Fuisz、Miguel Brandao、Josh Engels、Lars Wagoner 和 Tomáš Turlík。也感谢尝试这些任务的基线人员,包括 Ameya Prabhu、Anthony Armstrong、Johan Wind、Konstantin Akimov(knst)和 Mohamed Osman。数据与经验来自 Epoch AI,尤其是 Benjamin Ou 和 Avery Yen 分享的 EBR-Bench 及其人类基线;Emery Cooper、Linh Chi Nguyen、Alex Kastner 和 Caspar Oesterheld 分享的 LMCA 数据集及留出集;Owen Cotton-Barratt 和 Lawrence Phillips 分享的 JudgmentBench;接受访谈的 NanoGPT 速度跑者 Larry Dial、Varun Srivastava 和 Deven Pietrzak;以及提供 Al Zimmermann 竞赛耗时估计的 Tomas Rokicki。起草过程中,Lukas Finnveden 给予了特别密切的参与,Alexander Barry 为项目反应理论的建模选择提供了建议。其他讨论、评审和技术与运营支持,原文致谢部分有完整名单。
中文正文覆盖这篇更新的综述到致谢。附录 A 到 O 还包含基线协议、时间范围诊断、MCI 方法与限制、生命周期成本估计、数据完整性和奖励黑客、能力增速变化、推理扩展、群体实验,以及若干任务族的细节。下面按原顺序附上附录中的图,中文不再逐一展开这些技术细节。要核对这些图里的拟合、稳健性和任务级结果,请直接看原文附录。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。
原文
AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。

媒体未能载入,请通过下方原文链接查看。
Page image: https://metr.org/assets/images/q4-2026-capability-update/1eb15f15412f8e86-og.png
AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure - METR
Megan Kinniment, Manish Shetty, Brian Goodrich, Matt Putz, Spruce Bondera, Rowan Huang, Nikola Jurkovic, Jonathan Gabor, Oscar Sykes, Rif A. Saurous, and Beth Barnes
DATE
November 20, 2026
Contents
-
Overview
-
1 Background
-
2 Methodology
-
3 Results
-
4 Robustness Checks
-
5 Discussion
-
6 Acknowledgments
-
Appendices
-
A Difficulty Estimation and Baselining
-
B Time Horizon Diagnostics
-
C MCI Modelling Methodology
-
D Limitations of IRT
-
E MCI Robustness Checks
-
F Estimating Lifecycle Cost for Tasks
-
G Data Integrity and Reward Hacking
-
H Changes in Capability Growth Rates
-
I Inference Scaling
-
J Swarm Experiments
-
K Al Zimmermann's Programming Contests
-
L Budget NanoGPT
-
M Making METR tasks
-
N Earthborne Rangers and Continual Learning
-
O Qualitative and External Evidence on Models’ Strengths and Limitations
Overview Copy link to this section
This post presents evidence from our capability evaluations of public models relevant to AI R&D acceleration, up to Opus 5.5 and GPT-6 Astra. We remain significantly uncertain about central questions, including (1) what capabilities are in practice necessary for drastic acceleration, and (2) how quickly relevant model capabilities are improving.
We think that current public evidence is insufficient to rule out drastic AI R&D acceleration in the next couple of years. By drastic AI R&D acceleration we mean a situation where capabilities improve so quickly that humans are no longer able to meaningfully understand, assess, and react to catastrophic risks posed by new AI systems.
This overview summarizes our main takeaways and their relevant sections within the post.
1 Background describes our motivation and aims, and how our thinking about capability evaluations has changed since the release of the original time horizons paper.
2 Methodology describes the different task sets used throughout the post, changes to human baselining methodology, and introduces our use of multiple score thresholds per task.
Figure 1: November 2026 update on AI capability measurement

3 Results contains the evidence for our main takeaways, which are:
- Models usually match or outperform skilled humans on our coding and optimization tasks (Section 3.A.1). When we recruited our own human baseliners, we typically failed to find humans that outperform the current frontier models. In a couple of cases where we have existing sources of many (10+) domain specialists attempting a task, models tend to place at a very high percentile but not the absolute top (Section 3.A.2).
- On our dataset, models perform somewhat worse on tasks that we estimate are costlier to turn into RL training environments, even after controlling for how long these tasks would take humans. However, models appear to be improving at similar rates on our lower and higher cost tasks (Section 3.B.1). We observe that various properties that can make an eval more realistic also make the eval costlier to make, run, and score. We use a model estimated measure we call ‘Lifecycle Cost’ to try to quantify this observed costliness. In our data, a doubling of lifecycle cost for a task of a given length imposes a difficulty penalty that is very roughly equivalent to halving the time horizon of the agent. Despite this, we see similar rates of progress on high and low cost subsets of our tasks, and we also see AI performance improving over time even on our most costly tasks.
- On our tasks which require choices in the absence of clear empirical feedback, AI agents appear to have worse judgment than human experts (Section 3.B.2). This matters especially when feedback is slow or the available tests provide limited evidence that a solution will generalize. For instance, in one task we found that compared to an expert scorer, model judges overestimate an output’s quality when the output is made by a similarly strong or stronger model.
- On the combined suite of TH1.1 and TH2, time horizons are doubling roughly every 3.6 months (Section 3.C.1). Since TH2 is more difficult, any given model has a lower 50% time horizon on this combined suite compared to on TH1.1 alone. 1 We find that horizon doubling trends are shorter on a subset of TH2 (the verifiable subset) when excluding TH1.1. This could be consistent with a recent acceleration in time horizon that is hidden in the combined trend (which includes both TH1.1 and TH2) due to TH2 tasks being harder for a given time horizon, but there are also other plausible explanations. The doubling time on the combined suite is similar to the trend recently reported by OpenAI on internal research tasks. 2
- When tracking public model capabilities on a composite metric of model ability over time, we do not see strong acceleration or deceleration in model ability over the public frontier with the models we tested. However, we do see uncertain evidence of developer-specific trend changes in some conditions (Section 3.C.2). We use an index (METR Capability Index, or MCI) to track frontier AI capabilities over the past few years based on IRT, a statistical modelling technique similar to the one underlying the Epoch Capabilities Index. Similar to other indices of this kind 3, we do not observe slope changes in the overall public frontier trend (on the public models we tested) but we do observe some developer-specific slope changes. 4 However, we are unsure of the validity of these findings due to their sensitivity to methodological choices.
4 Robustness details various checks we did on our results, such as checking for reward hacking and refusals. We also investigate the effectiveness of agent swarms as an elicitation technique and find that they seem helpful on verifiable tasks with a clear metric and fast feedback (compared to best-of-k 5), and have unclear or negative effects on our other tasks.
5 Discussion describes open questions for assessing AI R&D acceleration.
The appendices give more detail on our methodology, report robustness checks for our analyses, and highlight individual tasks with notable model behavior.
1 Background Copy link to this section
In this post we focus on the measurement of capabilities relevant to accelerating AI R&D. A particularly salient uncertainty for AI R&D relevant capability evaluation is whether strong performance on ‘well-specified’ tasks alone is sufficient for drastic AI R&D acceleration, or whether other capabilities are needed. We aim to present evidence that is informative regardless of the reader’s view on this question.
Historically, METR used our ‘time horizon’ metric as a way to measure the SWE-related agentic capabilities of AI systems. 6 Time horizon measurements were useful for risk assessments because (1) We considered the capabilities we were measuring with our time horizon suite to be necessary to pose catastrophic risk, (2) Our time horizon measurements were substantially grounded in human reference data, which made it easier to relate to risk relevant thresholds, and (3) measured capabilities were far from possible danger, which allowed us to argue these systems posed negligible catastrophic risk with reasonable confidence. For example, see METR’s evaluation of GPT-5.
Much has changed since the original release of time horizon, and this has led to various changes in our approach and thinking:
- Public models have gotten a lot better at many tasks: When time horizon was first released in March 2025, the best models had a measured time horizon of around 1 hour (Sonnet 3.7). Within a year, models had saturated the TH1.1 suite, which includes tasks up to 30 hours. In many domains, it is now challenging and expensive to find human baseliners who outperform the models. 7
- Public models have strong enough capabilities in areas like coding and hacking that we no longer think we can construct an argument that risk is very low due to incapability on verifiable SWE-based tasks alone: For instance, public frontier models can hack services, 8 9 10 reason about the mechanics of how they are graded 11, and bypass restrictions set by sandboxes. 12 As a result, in order to make claims about risk from capabilities, we need to either (1) engage more with the specifics of the capability profile of the agent, and the capabilities required for a specific threat model, or (2) find other necessary broad agentic capability limitations that bound risk.
- We are increasingly concerned that our easy-to-run, containerized evals could overestimate real-world capabilities, due to similar RL environments becoming common in training. Both AI developers and evaluators have a shared desire for quantitative, cheap signal about AI capabilities (developers for training, and evaluators for measurements). As a result, we think evaluations often share much in common with the RL environments used to train models. This is a problem for evaluators because we often want to use our evaluations as a proxy for some other, harder-to-measure, capability of interest. 13 When these proxies (or nearby tasks) are trained on, they are liable to overestimate capabilities in these areas of interest. 14 We are somewhat concerned that AI developers' efforts to train on available cheap, quantitative capability signal mean that evaluators have relatively less uncontaminated signal to use for generalizable measurement of capabilities of interest.
These changes in the landscape have led to various modifications in our approach to capability evaluation. In summary:
- Human baseline and time estimation changes: Because our new tasks are harder and we aim to assess model capabilities relative to AI R&D lab employees, we recruited more experienced human baseliners than the STEM graduates in our previous work. The required expertise and time commitment made recruitment difficult, so we collected substantially fewer human baselines. We also allowed human baseliners to use Sonnet 5 for implementation. We did this to help with recruitment and try to avoid slowdowns associated with baseliners being used to coding with AI assistance. 15 Human baselines are directly used for only 9% of our estimates for how long it takes a human to complete the task. For tasks without successful human baselines, we use an estimation procedure informed by public records, experts, and extrapolation that we detail in Appendix A: Difficulty Estimation and Baselining.
- Task coverage and multiple score thresholds per task: To use our limited resources well we attempted to make more specific evals that targeted capabilities that we considered relevant to AI R&D relevant strengths and weaknesses. For instance, optimizing metrics, logic puzzles, and judgment-loaded tasks. As a result, the new tasks we have developed don’t cover as broad a range of general SWE work as the TH1.1 suite. Additionally, many of our new tasks now have multiple score thresholds, instead of just one. This allows us to extract more information from each task, at the cost of some of our analysis becoming more complicated, 16 see Section 2 for more information.
- Investigation and handling of suspected systematic measuring issues: As mentioned above, we are concerned about systematic measurement issues related to similarity of our evaluations and RL environments. We attempt to mitigate this problem by using more manually scored tasks and qualitative evidence, or otherwise developing evaluations that we expect to be less similar to the environments AI developers train on, while still being related to the capabilities we’re interested in.
2 Methodology Copy link to this section
We discuss results from several tasks and experiments, many of which have their own specific methods, budgets, and affordances (e.g., GPUs, internet access, etc.). This section describes shared methodology.
Agent Setup. For all tasks, we evaluated models using our standard ReAct-style scaffold which amongst other tools, has bash, python and image-viewing tools. 17 We run models with xhigh or higher reasoning effort (a setting that controls how much the model reasons before acting), and the highest available reasoning setting for models older than Feb 2026. Token and time limits were chosen per task with the aim that agent performance reached a plateau in further inference scaling, though this aim wasn’t always achieved, see Appendix I: Inference Scaling for token scaling curves.
Task Suites. Our results cover three distinct task suites:
- Time Horizon 1.1: 222 tasks including tasks from SWAA, HCAST, and RE-Bench.
- MirrorCode: 75 18 CLI software replication tasks.
- TH2: 36 tasks grouped into 12 task families 19 with two broad subsets:
a. TH2 verifiable: 8 families covering optimization, ML, puzzles, and games. b. TH2 hard-to-verify & judgment: 4 families covering conceptual reasoning, judgment, and hard-to-verify research.
Data Copy this chart's data
Figure 2: TH2 tasks are harder for AI agents and estimated to take humans longer

New tasks sets. The tasks we created sets span two kinds:
- Verifiable: In these tasks, agents work towards a well-defined objective that is scored automatically, such as an optimization target, score, or accuracy on held-out inputs. They usually measure their progress against the objective while working, although in some tasks they are required to build proxy metrics.
- Hard-to-verify & judgment: In our hard-to-verify tasks, agents work on an open-ended research problem with no cheap check for success or progress, and typically humans or a model given a human-written rubric judge final outputs. These research tasks are mostly based on our own work at METR. We also created more judgment-loaded tasks. By judgment, we’re broadly referring to the ability to make good choices in the absence of clear and fast empirical feedback. 20 In these tasks, agents are typically given an artifact (e.g., another agent’s transcripts on a task) and asked to judge its quality for some purpose.
The table below summarizes the family of tasks in each set.
| Task Set | Task Families |
|---|---|
| TH2 verifiable | AZsPCs (15 tasks): Solve heuristic optimization problems from Al Zimmermann’s Programming Contests. |
| Budget NanoGPT (2 tasks): Optimize a language-model training script based on the NanoGPT speedrun. | |
| Minecraft Bot (3 tasks): Build an autonomous bot that beats the game. | |
| Reverse Conway (2 tasks): Find a starting board that evolves into a target pattern in Conway's Game of Life. | |
| Train a Program (1 task): Train a neural network to reproduce the I/O of a given program. | |
| Earthborne Rangers (2 tasks): Play a card-based game that involves managing resources. This task is a lightly adapted version of Epoch’s EBR-Bench. | |
| Slay the Spire (4 tasks): Play a roguelike game through a text interface. | |
| Dominion-like Game (2 tasks): Play a deckbuilding game. | |
| TH2 hard-to-verify & judgment | Making METR Tasks (1 task): Build an evaluation task that METR would find informative. |
| Investigate Wiki Incident (1 task): Reconstruct an investigation into AI agents colluding on a wiki, based on MessageBoardAuditBench. | |
| LMCA (1 task): Rate critiques of philosophical arguments, scored by agreement with expert ratings, from Cooper et al. | |
| Judge Run (2 tasks): Score an agent’s run on Making METR Tasks with a rubric. |
Scoring. Unlike our earlier suites, TH2 includes some tasks that are manually graded by humans (Making METR Tasks) or model judges (Investigate Wiki Incident). 21 In Time Horizon 1.1, continuously scored tasks are binarized via a task-specific threshold chosen to represent human performance. For our new tasks, where possible, we instead use multiple score thresholds based on intermediate human performance on the task. We define these thresholds as (score, time) points: each point is the time taken by an expert human to reach a corresponding score.
We defined (score, time) points for 34 of the 36 new tasks using a mix of successful human baselines (3 tasks) and researcher estimates of score given time (31 tasks). Despite our best efforts to recruit strong human baseliners, sometimes our baselines fell short of what we estimated an AI R&D expert (e.g., lab employee) could achieve. In such cases, we used our estimates when we could make reasonable arguments for their feasibility. Our estimates usually draw on records of real human work, e.g., NanoGPT speedrun records, and in some cases experts or our own experience. Where a task had more than one baseline, we used the points of the highest-scoring baseline. See Appendix A: Difficulty Estimation and Baselining for more details. We include robustness checks on how sensitive the time horizon trends are to our estimates in Appendix B: Time Horizon Diagnostics.
Run coverage. On both TH1.1 and our new suite TH2 we typically ran 15 models: GPT-4o, o1, GPT-5, Opus 4.5, Opus 4.6, GPT-5.4, Opus 4.7, GPT-5.5, Opus 4.8, Fable 5, GPT-5.6 Sol, Opus 5, Fable 5.1, GPT-6 Astra and Opus 5.5. Where we refer to using the MirrorCode suite of tasks, we use older runs that do not cover the latest models. 22
Lastly, in this post we use an IRT methodology to summarize agent performance on a single scale and deal with uneven run coverage. We describe this methodology next.
METR Capability Index (MCI) Methodology Copy link to this section
In some parts of our analyses, we want a common scale for model capabilities and task difficulties. Even though the capability profile of models is often ‘jagged’ compared to the human capability profile, model capability profiles are usually quite similar between models. This presents us with an opportunity to roughly summarize their capabilities with a single latent factor. Item Response Theory (IRT) models are one way to fit data to find these latent ‘strength’ factors. The METR Capability Index (MCI) is an IRT model that is very similar in spirit to the Epoch Capabilities Index (ECI). However, unlike the ECI, which aggregates model performance across public benchmarks, our model treats the individual task as its unit of analysis and uses tasks we developed ourselves.
Notably, unlike a ‘time horizon’-like axis, IRT models do not require calibration against human performance. The ability axis reflects models’ patterns of strengths and weaknesses across tasks, rather than those of humans.
For each of our tasks, we define a scale in [0,1], where 0 corresponds to failure and 1 corresponds to full score 23, and fit a two parameter logistic for the score an agent (m) will get on a task (t):
[p_{mt}=\sigma (\alpha_{t}(\theta_{m}-\beta_{t})),\quad \sigma (x)=\frac{1}{1+e^{-x}}]
where (p_{mt}) is the expected score agent (m) will get on task (t), (\alpha_{t}) is the task discrimination, (\beta_{t}) is the task difficulty, and (\theta_{m}) is the agent ability. When an agent’s ability (\theta_{m}) is equal to a task’s difficulty (\beta_{t}), the agent is expected to achieve 50% of the maximum possible score. The slope parameter (\alpha_{t}) captures how quickly the expected score changes around that 50% point.
To fit this two-parameter logistic, we maximize the fractional-logit log-likelihood function. We also apply weightings to, for instance, upweight based on the number of runs we have on a task, downweight large numbers of similar tasks, and downweight ‘noisy’ tasks where our observed data is not well fit by our model. We generate 95% confidence intervals using hierarchical bootstrap across task families and runs. For full IRT model and fitting details, see Appendix C: MCI Modelling Methodology. We also do various robustness checks of our model, which can be found in Appendix E: MCI Robustness Checks.
3 Results Copy link to this section
In this section we’ll cover our results and other collected evidence. Section 3.A covers AI R&D relevant model capability strengths, Section 3.B covers model limitations, and Section 3.C covers capability trends.
3.A AI R&D Acceleration Relevant Capabilities Copy link to this section
We find that current public frontier models have extremely strong capabilities on optimization tasks, often outperforming skilled humans. Generally, these tasks have a clearly defined goal that is cheaply checkable. Usually the solution has few required criteria or constraints other than those expressed by some score, and often this score is observable to the agent.
Section 3.A.1 describes evidence that models outperform our humans on our hard but well-defined coding and optimization tasks. Section 3.A.2 describes model performance relative to top experts in places where we have the data to do so.
Appendix O: Qualitative and External Evidence on Models’ Strengths and Limitations also describes external evidence that model strengths in this area extend beyond just benchmarks.
3.A.1 Models often match or outperform skilled humans on coding problems and metric optimization with strong feedback signals Copy link to this section
We see strong performance from models on TH1.1, our old (primarily verifiable) sets of tasks, as well as on the verifiable portion of our TH2 tasks. Current public frontier models have an estimated time horizon of up to 27 hours on the verifiable subset of our TH2 tasks.
We also found that current public frontier models were able to beat the human baseliners that we recruited on the 3 tasks we attempted to acquire human baselines for. Since we want to measure whether models are able to replace AI developer employees, we aimed to acquire baseliners of a similar skill level. We recruited people with specialist knowledge in the task domain. For instance, for Train a Program our best external baseliner is a former International Grandmaster on Codeforces and won the 2020 Kaggle ARC-AGI competition. For Minecraft Bot and Reverse Conway, our best-performing baseliners were somewhat less credentialed, but had compensatory specialist knowledge or experience. We paid $200-350/hr and spent a total of roughly $90k for 7 attempted baselines.
Overall, we are uncertain about the comparability of our baseliners to AI R&D lab employees. We would guess that our baseliners’ skill within the task domain probably didn’t match the level of skill that typical frontier AI researchers have within their own domain. On the one hand, our baseliners were experienced, and often had specific experience relevant to the task. On the other hand, AI developers offer very high compensation, and have recruiting teams optimizing for finding highly talented individuals. Additionally, our tasks required dozens of hours of commitment, which plausibly selected against potential participants with full-time employment, and may have led to some adverse selection effects. For more details on human baselines see Appendix A: Difficulty Estimation and Baselining.
Various other work also observes high capability on these kinds of tasks. For instance, on MirrorCode, where agents reimplement software by checking against a full test suite, Opus 5.5 solves 77% of medium to large programs that would take human experts weeks to months to do on their own. GPT-6 Astra recently scored 98% on FrontierMath Tier 4, a benchmark of research-level math problems designed to take a strong mathematician specializing in that domain 1-2 weeks.
3.A.2 Where we have more human data, models outperform most humans on verifiable optimization tasks, but do not always dominate top experts. Copy link to this section
For two of our TH2 verifiable task families we have a larger pool of pre-existing human data to compare model performance against. On these two task families, we observe AI agents had high percentile performance but were generally unable to beat the very top performing humans.
Data Copy this chart's data
Figure 3: Current public frontier models outperform most humans (but not always the very top experts) on tasks with a large pool of participants

On the left, we show where model solutions rank against human participants in AZsPCs contests. On the right, we show model’s speedup over record #78 ranked among speedups of human records #79 to #92. For AZsPCs, models had 48 hours on a 28-core machine, and participants had about three months. For Budget NanoGPT, models had 48 hours on 8xH100s, and contributors typically reported 16 to 40 hours per record.
The first of these two task families was 15 Al Zimmermann contests, multi-month long optimization competitions. We find the median placement of the strongest models (Fable 5.1, Opus 5.5 and GPT-6 Astra) is the 96th to 98th percentile of human participants (97 to 413 participants per contest). We estimate that a dedicated human participant might spend between 90 hr and 180 hr over the course of the competition. We think the top of the human participant pool is quite strong. 24 In the median contest, roughly 4 humans beat the strongest models we tested, see Figure 3(a).
The second of these two task families was Budget NanoGPT speedrun, where we run the AI agent with a comparable amount of compute 25 to what human speedrunners typically use. Agent solutions (all starting from record #78) and human records aren't perfectly comparable, since each human record builds on previous records. However, interviews with contributors suggest that each human record took a similar amount of labor and resources, comparable to one of our agent runs. So we compare each agent run’s speedup to the speedup of individual human records, and believe this comparison is rough but still informative. Also note that recent "human records" can be AI-assisted (sometimes heavily so), and that we only include human records set up to Aug 2026.
On Budget NanoGPT speedrun, we find that fully autonomous AI agents outperform many competent speedrunners, achieving better speedups whilst costing less. However, top human contributors still can reach much larger speedups than AI agents, see Figure 3(b). The best AI agent run (of 2 to 5 runs per model) reached a 15.7% speedup over our starting record in 48 hours. The author of a recent unusually strong record (#92), with AI assistance for coding, estimated spending about 60 hours for a 41% speedup over the previous record.
For more detail on each of the results see Appendix K: Al Zimmermann's Programming Contests and Appendix L: Budget NanoGPT.
3.B AI R&D Acceleration Relevant Capabilities Limitations Copy link to this section
In this section we present evidence that current public frontier models have capability limitations that may prevent them from drastically accelerating AI R&D via automating research work.
In Section 3.B.1, we introduce Lifecycle Cost, a rough proxy for how costly a task would be to build and use as an RL training environment. We find that on our dataset, models perform worse on tasks with higher Lifecycle Cost, even when controlling for task length.
Section 3.B.2 describes evidence of limitations related to performing well on tasks where it is difficult to get clear empirical feedback about the quality of the solution. In these tasks, agents need to create their own metrics of quality, handle tradeoffs, or exercise judgment.
Appendix O: Qualitative and External Evidence on Models’ Strengths and Limitations also describes external evidence on weaknesses in the context of AI-related research.
3.B.1 On our dataset, models perform worse on tasks that we estimate are costlier to turn into RL training environments, even after controlling for task length. Copy link to this section
As discussed in Section 1, the evals that are METR’s current primary way of measuring model capabilities are likely to have much in common with the RL environments that companies use to train their models. This is because both METR and AI developers want to be able to cheaply, easily capture signals about model capabilities. As RL has become a larger share of training, we have become more concerned that this overlap causes systematic measurement issues.
To try to quantify how difficult a task is to make into an RL environment, we created a very rough proxy measure which we call Lifecycle Cost. This is the cost for us to develop a containerized environment for the task, and then to run and score the task 100 times. 26 The goal with this quantity is not to reflect the ‘real cost’ of RL environments per se, but rather to have some rough quantitative metric of relative costliness.
Table 1 gives examples of properties that we have found make it more expensive to create highly realistic tasks, which Lifecycle Cost aims to capture.
Figure 4: Properties that make it more expensive to create realistic tasks

Table 1: Lifecycle cost factors
| Cost factor | Example |
|---|---|
| Manual scoring | In many cases, programmatic scoring would overly constrain the task, and models may be unable to properly judge the quality of the output. In these cases, we can use humans to score outputs. This is especially costly when the human needs to be an expert. Examples include Making METR Tasks and Sunlight. 27 |
| Compute-per-run | Some of our tasks involve using GPUs or other compute infrastructure, which has an associated cost. In TH2, both Budget NanoGPT and Train a Program use GPUs. |
| Serial time | Evaluations that take a long time to complete are generally more difficult to construct and run. Long tasks tend to be more prone to technical issues and length often compounds other cost factors, such as monitoring cost. 28 Our longest evaluations in TH2 verifiable tasks are run for 48 hours. |
| Token costs | Tasks where models use more tokens are costlier to run. The token cost of an evaluation is often (but not always) correlated with the serial time needed to run the task. 29 |
| Action monitoring | Sometimes models behave in ways that can cause real world harm. METR monitors certain high-risk evaluations to prevent this. This increases token costs, and often requires human researcher time for approvals. Further details about METR’s monitoring setup are available in this post. |
| Environment complexity | Environments which offer more components or opportunities for interaction are generally harder to make. It is therefore often desirable to ‘import’ a complex environment from some existing artifact (e.g. a game, or a dataset). The process of importing the task whilst ensuring it is fair is generally harder for more complex artifacts as there are more opportunities for bugs, and increased labor cost of QA. Another obvious route for introducing environment complexity is allowing the evaluation to interact with the ‘real world’, which we discuss below. |
| Interaction with the real world | Allowing internet access, and particularly interacting with the real world (e.g. making money on some prediction market) is a potential way to incorporate lots of realistic complexity into a task. However, real world interaction can increase Lifecycle Cost by increasing monitoring requirements and QA costs associated with guarding against unintended solutions. These tasks also typically have more issues with noise (lack of reproducibility), have shorter lifetimes (the internet and world change over time), and involve resources that require some evaluator overhead to manage (e.g. cryptocurrency, credit cards, accounts). |
| Implicit generalizability requirements | In real work, there is often some gap between what measures are readily available to iterate against, and the ultimate requirements applicable to the solution. For example, it is easier to iterate against the currently known feature requirements of an application, and its related tests, than to choose an architecture which will be easy to adapt to future requirements and preclude harder-to-notice failure modes. Introducing these generalization requirements in a fair way usually requires giving agents more context on the wider situation in which its solution will be used, which takes additional development time. Additionally, the ultimate requirements on a solution are often most naturally exercised on longer timescales, which can make scoring harder. 30 |
| Making the task robust to reward hacking, and scoring sensitive to the capabilities of interest | It is often the case that when we first make a task, oversights in task design mean that agents can score highly via methods that do not test the capability of interest. Despite many of our tasks containing instructions to follow the intended spirit of the evaluation, we still find that we need to put in a lot of ‘hardening’ effort to prevent reward hacking or otherwise ‘cheesy’ solutions. This increases implementation and QA costs. This also increases running costs, since we typically check our runs for reward hacking each time they are run. 31 |
| Use of experts | Some tasks require specialist knowledge and time to create, run or score. For example, datasets created by experts (LMCA), and evaluations that require expert scoring (Making METR Tasks, Sunlight). |
| Human interaction during the run | Tasks which require human participation (e.g. in a game, or in an adversarial cyber challenge). Wall clock, coordination, and labor costs are significant for such tasks. 32 (None of our TH2 tasks include this). |
We estimate Lifecycle Cost for our tasks with an LLM judge (Opus 5.5). This judge is given the task code (including the README, instructions, and scoring), and asked to estimate various costs associated with the task like e.g. development, QA, and scoring costs. We also amortize shared development costs over families of similar tasks. We iterated on the prompt for the judge such that the judge’s estimates roughly matched our own for about 10 tasks across domains and cost ranges. 33 While we expect our prompt to generalize reasonably well to tasks outside ours, some choices like which phases we break cost into, how much we assume AI speeds up task development, labor rates, etc. reflect our own experience. For full details of how we estimate Lifecycle Cost see Appendix F: Estimating Lifecycle Cost for Tasks.
We think there are various limitations to Lifecycle Cost as a proxy for ‘difficulty to make a suitable RL environment’, and also to using ‘difficulty to make a suitable RL environment’ as a predictor of model performance broadly:
- RL environment creation difficulty doesn’t account for differing demand in different capability domains. For example, we might assume that AI developers are more interested in creating models that are good at coding, but much less interested in making models good at deception and sabotage.
- RL environment creation difficulty doesn’t account for cross-task generalization gains, or the amount / degree of task coverage needed for generalization within a domain.
- Our LLM judge is calibrated to our own estimates of Lifecycle Cost, which are informed by METR’s constraints. Labs are likely to have different constraints to us.
- Costs are likely to change as model capabilities improve. What is costly today may not be costly in a year, or a few months. For instance, judging according to a rubric may have been costly to do in 2023, but model graded rubrics are common today. In Table 1 we give examples of factors that have, up to now, increased Lifecycle Cost for us. We are not claiming that these are fundamental barriers, or even that these will be relatively costly in the future.
Figure 5 shows a scatter plot of human time-to-complete vs Lifecycle Cost for our tasks. For TH2 tasks, Lifecycle Cost is estimated only for the highest score threshold. On our dataset, human time-to-complete and Lifecycle Cost are quite correlated (R 2 = 0.82). 34
Data Copy this chart's data
Figure 5: Lifecycle costs are correlated with the estimated human completion times, for tasks in our dataset

When we control for this correlation between time-to-complete and Lifecycle Cost we find that (for a given length) tasks with higher Lifecycle Cost tend to be somewhat more difficult than what would be predicted by their time-to-complete alone (R 2 = 0.14), see Figure 6. 35 36
Data Copy this chart's data
Figure 6: Tasks we estimate would be costly to turn into RL environments tend to also be harder for models

In Appendix F: Estimating Lifecycle Cost for Tasks we show plots using all (score, time) points per task where we observe a similar but weaker effect of lifecycle cost, and more associations with task time.
However, our data is noisy and could be affected by a few things. For example, many of our task times are estimates, and if they are systematically wrong, part of this effect could come from error in task time. That said, it’s somewhat reassuring that when we use models to estimate task time instead of human estimates/baseline, we also observe a similar trend. Another reason could be that this effect is driven primarily by a few high cost tasks. In the plot above, we account for this by filtering such outlier tasks. More ablations are in Appendix F: Estimating Lifecycle Cost for Tasks.
When we subset our tasks by high and low Lifecycle Cost, controlling for task length, and plot time horizons over release dates, we find that models perform worse on the higher Lifecycle Cost subset of our tasks, see Figure 7. On average, the higher Lifecycle Cost subset is 2.9x as expensive as the lower Lifecycle Cost subset, and models perform 2.9x worse in terms of time horizon, equivalent to 1.5 doublings.
Interestingly, compared to the lower lifecycle cost subset, model performance on the higher lifecycle cost subset is closer to performance reported by OpenAI on their own internal research usage. 37 We don’t have many task distributions to compare here, but we think this is at least an encouraging sign about our ability to create evaluations that are directionally more predictive of performance on ‘actual’ AI R&D tasks. It also seems like some (weak) evidence in favor of lifecycle cost being correlated with factors of difficulty that make ‘real’ AI R&D internal research tasks more difficult for models than some kinds of evaluations.
Data Copy this chart's data
Figure 7: Models have lower 50% time horizons on TH1.1 and TH2 tasks that we estimate are costlier to make, even controlling for length. The costlier subset is closer to the reported difficulty of OpenAI research use.

See Appendix F: Estimating Lifecycle Cost for Tasks for how we split tasks by lifecycle cost controlled for time.
On our data, doubling times on the higher cost subset are similar to (and very slightly shorter than) doubling times on the lower cost subset. However, our task set spans a restricted range of relative lifecycle cost so it is possible that for sufficiently high relative lifecycle cost doubling times are longer (or alternatively, are shorter).
If we were to take seriously that lifecycle cost was a good proxy for realism, then differences in doubling times would be consequential for forecasting the impact of AI systems:
If higher lifecycle cost tasks actually have a slower growth rate, and relatively high lifecycle cost tasks are required, then naive extrapolation could predict that drastic AI R&D acceleration happens much later.
On the other hand, if there was no difference in growth rate at different (relative) lifecycle costs, then this would imply that each doubling in relative lifecycle cost simply delays trends by some number of months. On our dataset, a doubling in lifecycle cost for a given length of tasks results in model performance reaching 50% success rate approximately 4.6 months later. 38 Note that we are highly uncertain about the lifecycle cost of tasks required to drastically accelerate AI R&D. They could potentially be many doublings more expensive than the typical task in our dataset. 39
Overall, we remain uncertain about the relative growth rates of high lifecycle cost vs low lifecycle cost tasks. For more information see Appendix F: Estimating Lifecycle Cost for Tasks.
Using a similar methodology to time horizon, we also estimated Lifecycle Cost horizon as the model-estimated lifecycle cost at which agents have a 50% chance to succeed at the task. We find that the lifecycle cost horizon of models is also steadily improving, though with a longer doubling time compared to the time horizon trend. See Appendix F: Estimating Lifecycle Cost for Tasks for the lifecycle cost horizon trend (Figure 51) and corresponding logistic fits per model.
3.B.2 On domains where the ‘true’ quality of a solution is hard for the agent to determine due to prohibitively long feedback loops or requirements that the solution generalize, models appear to have worse judgment than human experts Copy link to this section
A property we consider to be somewhat captured within lifecycle cost is the verifiability or ‘fuzziness’ of tasks. Tasks that are ‘fuzzier’ tend to be harder to automatically score, or more faithfully scored in ways that are too resource intense for a typical evaluation context. These ‘fuzzy’ tasks are typically taken to load on skills like ‘taste’ and ‘judgment’, and often include ‘research’ as a central example. We think that making a binary distinction between ‘fuzzy’ and ‘verifiable’ tasks can be somewhat misleading. Many tasks often categorized as ‘fuzzy’ are in fact verifiable in some way, given sufficient time to let the consequences of a given solution play out. For instance, making a ‘good app’ for some purpose might be considered a fuzzy task, but given enough time and exposure to use, we could eventually observe more objective and faithful measures of quality e.g. number of users or ratings. One way to operationalize this is in terms of the general costliness of assessing the quality of a solution in the long run. 40 Tasks where it is expensive to actually ‘play out’ the whole process to a point where long-run quality can be assessed (e.g. deciding a research agenda), are costlier-to-verify. We therefore expect these ‘fuzzier’ or ‘costlier-to-verify’ tasks to have higher lifecycle costs than typical for their length.
In this section we discuss evidence related to the ‘judgment’ capabilities of models and provide a few other examples of weakness related to managing one’s own feedback loops. As defined in Section 2, by judgment, we’re broadly referring to the ability to make good choices in the absence of clear and fast empirical feedback. This is often driven by strong argument and reasoning, or instilled via experience and expressed through the careful application of learned heuristics. In practice, this can short circuit otherwise necessary and expensive feedback loops. As a result, ‘judgment’ is particularly helpful on ‘costlier-to-verify’ tasks.
It is unclear to us the degree to which drastic acceleration of AI R&D will require judgment or related capabilities. We discuss this further in Section 5. However, we would consider weakness in ‘judgment’ and related capabilities to be reasonable candidates for what most delays drastic AI R&D acceleration.
As one measure of these domains, we ran LMCA, 41 a benchmark where models need to rate critiques of philosophical arguments. Models have shown steady improvement, and many have agreement above the estimated agreement level of expert humans who spent 5 minutes on the task, but all public models are still below the estimated agreement level of expert humans spending 15 minutes. 42
AI agents now build borderline-useful evaluation tasks, but not yet ones we consider especially informative Copy link to this section
We made a task designed to test model ability to automate making our evaluations. The agent is asked to create a new task for our evaluation suite which is informative for METR’s assessment of model capabilities. None of the tasks agents produced were sufficiently useful to us to warrant inclusion in this post. 43
Data Copy this chart's data
Figure 8: AI agents now build borderline-useful tasks, but not yet ones we consider especially informative

The models we tested generally favored making automatically scored optimization tasks, even when (in follow-up experiments) instructed to make tasks where "success is sensitive to decisions which have poor short term feedback." Agents did not independently try to improve their understanding of what tasks METR would find most useful, despite internet access. We did not observe agents produce novel threat modeling, or construct tasks based on particularly insightful hypotheses about model capabilities. 44
AI judges are overgenerous when judging tasks made by models stronger than themselves Copy link to this section
We observed related weaknesses when models are asked to judge the quality of other model-made tasks. When asked to score tasks according to how useful they’d be for us, their agreement with our own ratings depends on the relative strength of the judge model and the task-creator model.
Frontier models show high agreement with our own ratings when the task is made by a weaker model than themselves, but give inappropriately high scores when the task is made by a stronger, or similarly strong, model, see Figure 9. 45 Opus 5.5, the judge that had the highest agreement with our human scorer, scores Astra’s tasks on average about 0.2 (out of 1) more highly than our human. 46
Data Copy this chart's data
Figure 9: AI judges are overgenerous on tasks made by stronger models

The most straightforward explanation for this is that models have poor judgment about what makes a good task. This poor judgment effectively caps the model’s performance during the task itself, because the model only has its own judgment to optimize against. With the end result being that when used as judges they systematically overestimate the quality of their own solutions, and those of stronger models. Another possibility is that strong models exhibit behaviors that cause weaker judges to overestimate their solutions. 47
Some caveats to our judgment related results:
Firstly, we think it is somewhat likely that models are underelicited on these kinds of problems. For example, in our setting and similar settings 48, we observe models significantly underutilize their provided token limits and affordances.
Secondly, we do see current public frontier models displaying good judgment (of unclear generality) on some ML tasks. For instance we observe that on days-to-weeks long ML evaluations with programmatic scoring, models are able to beat our human baseliners. This includes tasks that we explicitly designed to be novel, and that we do not believe to have been trained on. In related work, P-Zero Research also finds strong performance on TasteVal, which measures the compute efficiency of agents' experiment choices in up to 120hr long verifiable ML optimization tasks. 49 And this performance seems to persist under resource limits matched to those that humans work under (as in TasteVal or our Budget NanoGPT results). However, it seems plausible to us that the judgment displayed by models on these tasks is in some sense less general than the judgment displayed by humans. It also seems plausible that there is no fundamental difference.
3.C AI R&D Acceleration Relevant Capabilities Trends Copy link to this section
3.C.1 Across TH1.1 and TH2 we find that capabilities are doubling roughly every 3.6 months Copy link to this section
Across our TH1.1 and TH2 tasks, we find that time horizons are doubling roughly every 3.61 months (95% CI 2.08-4.94), or 110 days, see Figure 10. This is similar to the previous TH1.1 doubling trend of 105 days. 50 On these tasks, naive extrapolation would imply that models reach a 50% time horizon equal to the longest score threshold on our task set (200 hours) in July 2027.
Data Copy this chart's data
Figure 10: Across TH1.1 and TH2 we find that capabilities are still increasing quickly, doubling roughly every 3.6 months.

In Appendix B: Time Horizon Diagnostics, we see similar results when noising our baseline estimates and when using model estimated times instead of our default time estimates. 51
When we estimate separate trends for task subsets, we observe a faster doubling time on TH2 verifiable tasks than on TH1.1 tasks. Our data is noisy, but for TH2 (verifiable) we see a doubling time of 1.65 months (95% CI 0.79 to 3.99), see Figure 11. This trend is faster than the combined TH1.1 + TH2 doubling time. 52
Data Copy this chart's data
Figure 11: When separated, time horizon doubling times on TH2 (verifiable) appear shorter than on TH1.1.

We also show trends on MirrorCode, another verifiable task distribution. Trends are fit to frontier models. Doubling time CIs are calculated by bootstrapping over models, families, and runs. Model CIs are bootstrapped over families and runs. In this figure we show a wider array of models, due to not having to constrain to data on which we have runs for every (model, task) pair across TH1.1 and TH2. Logistic fits are given in Appendix B: Time Horizon Diagnostics.
This faster doubling time could be taken as evidence of an acceleration in model capabilities in 2026 (at least as measured by time horizon, on these tasks). However, we think it is worth noting that we already observe similarly quick doubling times on MirrorCode in the second half of 2025 (another, easier, verifiable distribution of tasks), when TH1.1 was still growing at a slower rate.
We are not sure what the cause of the faster doubling time on TH2 (verifiable) and MirrorCode is. One possibility is a general acceleration in capabilities measured by time horizon, as mentioned above. In principle, such an acceleration could be hidden in the combined set by an artificial deceleration from switching over to TH2, which is generally a harder set of tasks at a fixed time horizon. 53 Some other possibilities include these distributions having data over a more restricted date range, or that tasks involved have properties that make them easier to improve at quickly, or combinations of these possibilities.
One possibility that we find plausible is that doubling times may tend to be shorter on less diverse sets of tasks. Essentially, when a set of tasks are very similar to one another, longer tasks might not be that much more difficult (e.g. if it is mostly more of the ‘same kind’ of work). Or alternatively stated, once models have learnt the main skills needed to do a kind of task, they are relatively quickly able to do longer versions of that kind of task. However, across a varied set of tasks, longer tasks are more likely to include different or multiple different kinds of work, such that models may overall tend to acquire the necessary abilities later. So relative lack of diversity may be a contributing factor towards MirrorCode and TH2 (verifiable)’s shorter doubling times. 54
Another possibility is that our selection for difficult tasks during development contributes to TH2 (verifiable) having a shorter doubling time. However, we would be somewhat surprised if this was the majority of the reason for the faster doubling time of TH2 (verifiable), since we e.g. kept tasks that were saturated during the suite’s development.
We also note that recently published data from OpenAI seems to imply a doubling time of 3.8 months when using their models on their own internal research tasks, which is similar to the 3.6 month doubling time we find over TH1.1 and TH2. 55
3.C.2 We do not observe a strong acceleration in capability across public frontier models, as measured by METR Capability Index (MCI) on our tasks. However, we do see uncertain evidence of developer-specific trend changes in some conditions. Copy link to this section
If AI R&D were already being substantially sped up, we might expect to observe an acceleration (i.e., a positive change in growth rates) in the capability of models over time (even if they continued to have a jagged profile). We see rapid progress on tasks when measured in terms of time horizons. Given the uncertainty we have about whether ‘task length’ 56 is a good metric of task difficulty, and the value of our time estimates, we also analyzed our data with MCI, an IRT model.
When we fit our MCI model (similar to metrics like Epoch's ECI, and Anthropic’s AECI) we do not observe an obvious acceleration over all public frontier models, see Figure 12. We believe this result provides moderate evidence against a general deceleration in model capabilities and weak to moderate evidence against a general acceleration over all the models we tested.
Data Copy this chart's data
Figure 12: When fitting the MCI model to our data, we do not see an obvious acceleration or deceleration in capabilities over time.

However, if we instead calibrate MCI and fit trends separately for each AI developer, we do observe some slope changes (i.e., increasing capability growth rates) in certain conditions. We find that slope changes on developer-specific MCI are generally comparable to those presented in METR’s Independent Assessment of AI R&D Acceleration at Anthropic in October 2026 and recent Anthropic system cards in the conditions that we are most able to reproduce, i.e. on public Anthropic models up to Opus 5.5. We are uncertain about the validity of these slope changes, since we find they are quite sensitive to various methodological choices. We discuss these AI developer specific slope changes and sensitivity to methodological choices in Appendix H: Changes in Capability Growth Rates.
Note that, when using this kind of IRT methodology to detect acceleration, we think it is worth bearing in mind some reasons that this method may ‘miss’ an actual acceleration. Firstly, it is possible that developers could be delaying public releases for longer as models get more capable, 57 which could cause us to not detect an acceleration. Secondly, MCI always has more uncertainty in the ability of the most capable AI model, 58 which we expect to delay our ability to detect an actual acceleration.
This methodology is also only able to detect acceleration in overall ability as measured by MCI, rather than acceleration caused by AI specifically. We do not know what the counterfactual trend would have been had AI developers not been using AI to help with their research. It’s entirely possible that without AI we would see a decelerating trend, and that AI use has in fact accelerated trends to the current level.
For more details on our methodology, see Appendix C: MCI Modelling Methodology, for robustness checks see Appendix E: MCI Robustness Checks, for additional discussion of limitations of our model and related IRT models, see Appendix D: Limitations of IRT.
4 Robustness Checks Copy link to this section
We did various robustness checks for our evaluation results:
To check that our token budgets for runs were large enough we looked at how performance scales with increasing tokens. For each task, we aimed to set a token budget where performance plateaus, though this was not always achieved. For more details, see Appendix I: Inference Scaling.
We also checked for reward hacks. Runs with reward hacks were assigned a score of zero. To do this we manually looked through a subset of traces, and ran model scanners over the runs (sometimes with instructions on what to look for by task, informed by reading through the subset). Most of the reward hacking we found was by near-frontier agents, 59 such as GPT-5.6 Sol, GPT-6 Astra, Fable 5, Fable 5.1, and Opus 5.5. For more details see Appendix G: Data Integrity and Reward Hacking and Appendix G: We find that models still often engage in reward hacking.
We excluded runs where classifiers blocked generation, or the model itself refused. This was quite rare and affected about 36 runs across both TH2 and TH1.1. We also performed some preliminary checks for sandbagging, see Appendix G: Refusals and Sandbagging for more.
Finally, we experimented with using “swarms” rather than our regular scaffold. We generally compared swarm runs to best-of-k (which typically performs better than a single very long serial run on these kinds of tasks). Though our data is noisy, we found that swarm scaffolds seem to increase performance over best-of-k controls on tasks with a clear metric to optimize. In other tasks, we found unclear or negative effects. In particular, we found that large swarms seemed harmful to performance when the task required creating a proxy metric for themselves. For more detail see Appendix J: Swarm Experiments.
These results lead us to believe that we may be somewhat undereliciting performance on our clean verifiable optimization tasks due to not using swarms. We expect that not using swarms on our other tasks is not a contributor to underelicitation. We generally found that swarms were very expensive to run; our swarm experiments used approximately $500k in tokens.
5 Discussion Copy link to this section
As model capabilities have improved, some methods and evaluations that we have previously used to upper bound risk have become less useful, or obsolete. Current public models now have very strong performance on well-scoped ML, cyber, and coding problems. Therefore, it is now more difficult to bound risk by pointing to some easy-to-measure and broad incapability.
Instead, our understanding of risk from model capabilities is more dependent on questions related to the specific details of model capability profiles, and facts about the world. We do not feel that we currently have good answers to many of the most important questions here. For instance:
- How much stronger are model capabilities on easier (or less-costly) to verify tasks, compared to harder (or costlier) to verify tasks? On our dataset, we observe weaker performance on the higher relative lifecycle cost subset of our tasks. However, the link between lifecycle cost and some notion of ‘realism’ remains very unclear, and we do not know if its associations with worse model performance apply to other task distributions.
- Are models getting better at harder-to-verify tasks more slowly? If so, by how much? On our dataset, we see similar doubling times on the high and low relative Lifecycle Cost subsets of our tasks. However, we could imagine that the trend could be different on tasks with much higher lifecycle cost, or that were otherwise more realistic.
- What capabilities are even necessary for drastic AI R&D acceleration? In this post we have largely avoided making claims about what is necessary and sufficient for drastic AI R&D acceleration, but this is a major source of uncertainty for us. It may in fact be possible to drastically accelerate AI R&D via easy-to-verify tasks alone. As a result, we think it is prudent to consider the possibility that AI may be able to drastically accelerate AI R&D before we see good performance on many ‘realistic’ tasks that may be necessary for the automation of ‘normal’ jobs. At the very least, AI developers are putting a great deal of active effort into getting their AIs to automate their own research, 60 which we might expect to bring forward automation relative to AI R&D’s ‘natural’ difficulty.
6 Acknowledgments Copy link to this section
We thank the following task developers, who helped with implementation, quality assurance and reward-hacking checks: Drew de Wet, Gerard Boxo, Martin Milbradt, Máté Csikós-Nagy, Pablo Romero, Prakrat Agrawal, Gábor Fuisz, Miguel Brandao, Josh Engels, Lars Wagoner, and Tomáš Turlík. We also thank our baseliners, who attempted the tasks we designed, including Ameya Prabhu, Anthony Armstrong, Johan Wind, Konstantin Akimov [knst], and Mohamed Osman.
We are grateful to those who shared datasets and expertise with us. Epoch AI, and in particular Benjamin Ou and Avery Yen, shared EBR-Bench and data from their human baseline study. Emery Cooper, Linh Chi Nguyen, Alex Kastner and Caspar Oesterheld shared the LMCA dataset, including its held-out set. Owen Cotton-Barratt and Lawrence Phillips shared JudgmentBench. The NanoGPT speedrunners Larry Dial, Varun Srivastava, and Deven Pietrzak spoke with us in interviews. Tomas Rokicki shared estimates of how much time he put into Al Zimmermann’s Programming Contests.
We thank Lukas Finnveden for especially close engagement with the drafting of this post, and Alexander Barry for advising on our IRT modeling choices. We thank Jeff Wu and Tao Lin for discussion of our task distributions, and Francis Rhys Ward, Alex Jackson, Titus Buckworth, and Jason Brown for discussion of evaluation choices. We thank Francisco Carvalho for task development and for early work on Elo-based capability metrics, and Niels Warncke for task development. We also thank the following reviewers for feedback on our drafts: Alexander Barry, Parker Whitfill, Tom Cunningham, Asa Cooper-Stickland, Tao Lin, Charles Foster, and Jean-Stanislas Denain.
We thank Mischa Spiegelmock, Paarth Shah, Dmitrii Calzago, Pip Arnott, Rafael Carvalho, Jack Misbach, Brandon Qi, and Rasmus Faber-Espensen for technical support throughout the project. We thank Bhaskar Chaturvedi, Jingyi Wang, Kris Chari, Parv Mahajan, Rae She, Stephanie Palla, Andrew Wei, and Wilder Seitz for operational support, and Charles Foster for help with communications.
Finally, we thank Nate Rush and Hjalmar Wijk for support in the early stages of the project.
Appendices Copy link to this section
- A Difficulty Estimation and Baselining
- B Time Horizon Diagnostics
- C MCI Modelling Methodology
- D Limitations of IRT
- E MCI Robustness Checks
- F Estimating Lifecycle Cost for Tasks
- G Data Integrity and Reward Hacking
- H Changes in Capability Growth Rates
- I Inference Scaling
- J Swarm Experiments
- K Al Zimmermann's Programming Contests
- L Budget NanoGPT
- M Making METR tasks
- N Earthborne Rangers and Continual Learning
- O Qualitative and External Evidence on Models’ Strengths and Limitations
A Difficulty Estimation and Baselining Copy link to this section
Our time horizon results, along with analyses that control for task length, depend on estimates of how long it would take a skilled human to reach a given score on a task.
For most tasks in our original time horizon suite, we measured this directly by having skilled professionals attempt the same tasks we evaluate models on, under the same conditions. We call these attempts human baselines.
Because our new tasks are far harder than those in our original suite, this was no longer feasible. It was often difficult to find qualified people who could commit the time, the cost of collecting these baselines was much higher, and models often outperformed even the experts we did recruit.
For tasks in the new suite, we use researchers’ estimates of the time needed to reach a range of scores. For most of the tasks we have evidence from humans doing the same or similar tasks, but usually not in a dedicated, self-contained setup. We ran formal human baselines for three of our twelve task families, though for two of them we also relied on other sources of data and on extrapolation, as we describe below.
The rest of this section covers our protocol for the formal baselines, how we recruited baseliners, and their results. We then explain how we produced human time estimates for each task.
Baselining Protocol Copy link to this section
Each baseliner worked in a container identical to the one an AI agent is given, with the same information and resources. We avoided giving the baseliners guidance beyond what agents receive, so that humans didn't have an unfair advantage.
Baseliners were allowed to use Sonnet 5 for implementation, but we asked them not to rely on it for high-level decisions. This differs from TH1, where baseliners mostly couldn’t use AI. 61 We allowed AI assistance partly because many people now use AI in their work, and some people didn’t want to do baselines without it because the work would be too tedious. Another reason is that letting baseliners use AI makes baselines cheaper to collect, as they can spend more of their time on difficult work that requires their expertise and less on routine work. We didn’t allow access to a frontier model to ensure that human baselines were still mostly driven by human performance.
We also let baseliners access the internet.
We didn’t give baseliners hard time limits, though we did share a rough estimate of how long we expected the task to take. We also didn’t give them target scores or an indication of what strong performance would look like. Throughout the baseline, we periodically checked their progress and decided whether to have them continue.
Baseliners were paid $200–350/hr. We didn't use bonuses for reaching performance milestones. However, the progress check-ins meant baseliners who wanted to keep working had some incentive to perform well.
The table below summarizes how our protocol compares to TH1.
| TH1 | TH2 | |
|---|---|---|
| Baseliner profile | Professionals with ~5 years' experience who passed a qualification task | Task-specific specialists |
| Pay | Typically $50–100/hr, plus performance bonuses | $200–350/hr with no bonuses |
| AI assistance | Generally not allowed | Allowed for implementation (Sonnet 5 only) |
| Internet | Generally allowed | Allowed |
| Target score given | Generally yes | No |
| Time limit | Up to 8h per task | No hard limit, with periodic progress reviews |
| What we measure | Mostly time to complete the task | Time to reach one or more score levels |
Recruiting Baseliners Copy link to this section
With these baselines, we aimed to compare model capabilities with those of AI R&D experts, ideally people working at frontier AI companies. Of the three tasks we baselined, only Train a Program is directly an AI R&D task. The other two test skills we think are closely related to AI R&D. For Train a Program, we aimed to recruit people with skills comparable to AI company researchers. For the other tasks, we looked for people with a similar level of expertise in the relevant domain. This made recruiting challenging, as these people often have demanding, well-paid jobs, so they may be too busy to complete a baseline.
Here is some background on the baseliners for each of the three formally baselined tasks:
- Reverse Conway. We had two baseliners for this task. One was on the second-place team in a closely related Kaggle competition, and the other was a PhD student working on SAT solving.
- Minecraft Bot. We had one baseliner for this task, who had previous professional experience building bots for Minecraft and League of Legends.
- Train a Program. We had four baseliners for this task. They all had backgrounds that made us expect them to do well, such as top results in programming or AI competitions or prior ML research. The baseliner with the highest score was a former International Grandmaster on Codeforces and the winner of the 2020 Kaggle ARC-AGI competition.
Overall, we’d weakly guess our best baseliners would do better on these tasks than a typical technical employee at a frontier AI company (since they have more relevant expertise), but they’d probably do worse than a typical frontier AI researcher in their own research area.
Baseline Results Copy link to this section
On all three tasks we formally baselined, the best model outperformed every human baseliner.
Data Copy this chart's data
Figure 13: Public frontier models outperformed every human baseliner on three tasks we baselined

That said, we think some aspects of our tasks are artificially unfriendly to humans, in ways that don’t reflect what it’s like to do this kind of work in more natural settings. This might contribute to humans underperforming models, since models are well-adapted to containerized task environments in general. One example is that on some tasks, a mistaken assumption about the setup or scoring can leave a baseliner with a much worse score, even if their underlying work is otherwise good. This seems to be less of a problem for models, likely because extensive RL training has made them very good at avoiding mistaken assumptions about these kinds of task environments. In future research, we think it might be worth giving humans some amount of feedback as they’re going.
Although models outperformed our baseliners on all three of these tasks, humans did better on some tasks not included in the final TH2 suite. On two social deduction game tasks, humans scored higher or the same as the models. On Sunlight, a manually scored task where models design an indoor setup to recreate sunlight and write a report on its optical properties, we think most people would prefer the human-written report, though agents’ submissions are better in other ways. 62
How we produced the human time estimates used in our analyses Copy link to this section
For each task, our researchers estimated how long a skilled human in the relevant domain with access to Sonnet 5 would take to reach each score. The table below shows where our estimates for each task came from.
| Task Set | Task Families (* = especially speculative estimate) | Evidence that informed our difficulty estimates |
|---|---|---|
| TH2 verifiable | AZsPCs | Email correspondence with a top competitor. Public data from human competitors. |
| Budget NanoGPT | The public modded-nanogpt record history. Interviews with the authors of several records. Effort estimates from METR’s Expenditure Horizon post. | |
| Minecraft Bot* | GitHub commit history of a public solution roughly equivalent to our task’s setup. One formal human baseline (more context below). | |
| Reverse Conway | Two formal human baselines. | |
| Train a Program* | Four formal human baselines. Two baseliners stopped after 1–3 hours, and only one scored above zero (but still less than the agents). One informal baseline by a METR staff member (more context below). | |
| Earthborne Rangers | Epoch’s study of 13 people playing a text-based version of the game, which differs slightly from ours. | |
| Slay the Spire | Public run histories from Slay the Spire streamers (Xecnar, Baalorlord, and TerrenceM). | |
| Dominion-like Game | One informal baseline. Public leaderboard scores from experienced players. | |
| TH2 hard-to-verify & judgment | Making METR Tasks* | Multiple informal 2-hour baselines. METR staff’s experience making tasks as part of our regular work. |
| Investigate Wiki Incident* | Two METR researchers’ time estimates, based on the reference solution. | |
| LMCA | The LMCA dataset’s main expert rater’s estimate of how long an average rating took. | |
| Judge Run | Time METR staff logged when manually scoring the same runs the agent is asked to judge. |
For some tasks, we wanted difficulty estimates for scores higher than any human had reached in the data we relied on. In these cases, we extrapolated by estimating how much more time humans would need to reach those scores.
For both Train a Program and Minecraft Bot, we assumed stronger human performance than the formal baseline data alone would suggest. We made this subjective decision because we think it’s likely the very best experts in each domain could reach higher scores than our baseliners, and because, as mentioned above, these tasks are artificially unfriendly to humans in some ways. 63 Specifically:
- Train a Program. We based our estimates on an informal baseline by a METR staff member, who scored 0.21 in about 10 hours, and extrapolated on the assumption that they would keep improving at a similar rate. This baseline isn’t directly comparable to the external ones, as the staff member had some prior knowledge of the task, including the test distribution, and used Fable 5 rather than Sonnet 5, though mainly for implementation rather than coming up with ideas. Even given this, we chose to base our estimates on it rather than the external baselines, because the staff member believed they could have done similarly well without these advantages and would have kept improving with more time. By contrast, our best external baseliner, who scored 0.042 after 45 hours, expected only a roughly 2x improvement given 3-10x more time, which would still be far below the best models.
- Minecraft Bot. Our estimates are based on AltoClef, a public bot that beat the game in 2021. We used the timestamps in its commit history to estimate how many hours its author had worked by the time the bot first reached each of our task’s checkpoints. Our external baseliner reached the first checkpoint in about the same time as AltoClef’s author (about 5.5 hours). After that, their graded score only improved slightly, reaching 0.055 after 160 hours. Our estimates assume that a top expert could keep progressing at a similar pace to AltoClef’s author.
Overall, we consider these estimates quite speculative. However, when we test how sensitive our time horizon trends are to changes in these estimates, we find the trends are fairly robust, see Appendix B: Time Horizon Diagnostics.
B Time Horizon Diagnostics Copy link to this section
In this appendix, we show diagnostics figures and robustness checks for our time horizon calculations.
Average Scores across Tasks Copy link to this section
In the chart below, we attempt to give an intuitive sense of how agents perform relative to human score-time points across our time horizons task suite. Each row represents a task from our new suite at a specific score threshold defined by a human (score, time) point. Each cell is a model’s average score on that task colored by how far it is from the score threshold, where red indicates performance below that row’s human score, and green indicates performance above that row’s human score.
Data Copy this chart's data
Figure 14: Average Agent Scores across Tasks

Each cell is a model's mean score on the task. Each row is one human (score, time) point, and color shows how far the model's mean score is above or below that human score. Tasks are grouped by family and sorted by IRT difficulty.
Logistic Fits for our Time Horizon Estimates Copy link to this section
Throughout this post we use the Time Horizon methodology on various task distributions and combinations of task distributions. Here, we show success rates and the logistic fits for them:
TH1.1 + TH2 Verifiable Copy link to this section
Data Copy this chart's data
Figure 15: Success Rate & Logistic Fits: TH1.1 + TH2 Verifiable Tasks

Each point is one human (score, time) point from the TH1.1 or TH2 Verifiable tasks, at the share of the model's runs that reach it, colored by that share.
Lower Relative Cost Subset of TH1.1 + TH2 Copy link to this section
Data Copy this chart's data
Figure 16: Success Rate & Logistic Fits: Lower Relative Cost Tasks across TH1.1 + TH2

Tasks with lower Lifecycle Cost controlled for task time (for humans).
Higher Relative Cost Subset of TH1.1 + TH2 Copy link to this section
Data Copy this chart's data
Figure 17: Success Rate & Logistic Fits: Higher Relative Cost Tasks across TH1.1 + TH2

Tasks with higher Lifecycle Cost controlled for task time (for humans).
TH1.1 Copy link to this section
Data Copy this chart's data
Figure 18: Success Rate & Logistic Fits: TH1.1 Tasks

Tasks every model shown was run on.
MirrorCode Copy link to this section
Data Copy this chart's data
Figure 19: Success Rate & Logistic Fits: MirrorCode Tasks

Tasks every model shown was run on.
TH2 Verifiable Copy link to this section
Data Copy this chart's data
Figure 20: Success Rate & Logistic Fits: TH2 Verifiable Tasks

Tasks every model shown was run on. Short TH1.1 tasks (under 10 minutes) are included to help constrain the fits.
Robustness Checks for Time Estimates Copy link to this section
In this section, we check how robust our results are to changing our estimates of human time-to-complete.
1. Noising human baseline estimates Copy link to this section
To test how much our trends depend on the exact task time estimates, we refit each trend after multiplying every task's time by random noise. We find that the doubling times stay close to the originals. This holds even for TH2, where we added the most noise because task times were largely based on researcher estimates rather than human baselines.
Data Copy this chart's data
Figure 21: Doubling time robustness to noised time estimates

We fit each bootstrap draw twice, once with our task times and once with noise on every task time. Tasks in one family get the same noise. For TH1.1 tasks, the noise size is from the original time horizon paper and is smaller for tasks with more human baselines. For TH2 and MirrorCode tasks, 68% of noised times are within 0.25x to 4x of our estimate. Draws with no valid trend are dropped.
2. Using model estimates of how long it takes humans to do a task Copy link to this section
In this check, we regenerate our main TH trend graphs, using model-estimates of human time-to-complete, instead of using estimates from human baselines and public-evidence extrapolation.
We find that model-estimates of human time-to-complete are similar to estimates we derive from human baselines and manual estimation (see Appendix A: Difficulty Estimation and Baselining). The main divergences come from estimating human time-to-complete for MirrorCode tasks.
Data Copy this chart's data
Figure 22: Model Estimated vs. Human Task Time (for humans)

As a consequence, we generally find similar trends on our trend graphs:
Data Copy this chart's data
Figure 23: TH1.1 and TH2 time horizon trend, with human and model-estimated task times

Left: task times from our human baselines and estimates. Right: each task's time is a model's estimate of how long a human needs to pass the score threshold. Trends are fit to frontier models. Doubling time CIs are calculated by bootstrapping over models, families, and runs. Model CIs are bootstrapped over families and runs.
Data Copy this chart's data
Figure 24: Time horizon trends by task set, with human and model-estimated task times

Left: task times from our human baselines and estimates. Right: each task's time is a model's estimate of how long a human needs to pass the score threshold. TH2 does not contain many short tasks, so to constrain its logistic fits we add short (<10m) TH1.1 tasks to TH2 in this figure. We also show trends on MirrorCode, another verifiable task distribution. Trends are fit to frontier models. Doubling time CIs are calculated by bootstrapping over models, families, and runs. Model CIs are bootstrapped over families and runs.
3. Using longer human time-to-complete estimates Copy link to this section
In this robustness check, we change time estimates where we thought it was possible we were systematically underestimating length.
We do two variants of this:
- Increasing the time estimates for Train a Program by 4.8-12.5x, and for Minecraft Bot by 2-4x for the (score, time) points beyond our best baseliner.
- Increasing the time estimates for all TH2 (score, time) points with scores above 0.5 by 4x.
Data Copy this chart's data
Figure 25: TH1.1 and TH2 time horizon trend under alternative time estimates

Counterintuitively, these alternate estimates reduce time horizon estimates and their CIs for recent models. We think this happens because when there are few points far out in time, the logistic fit has little information to determine when it should fall to low values. When this happens, the logistic fit flattens out, pushing point estimates longer and widening the CIs. 64 So moving some failure points to be further out along the x-axis (as increasing these time estimates does) can actually steepen the logistic fit. 65
Data Copy this chart's data
Figure 26: TH1.1 and TH2 logistic fits under alternative time estimates

Each panel refits the same runs after changing only the estimated human times of score thresholds.
| Pooled TH1.1 + TH2 50% time horizon (95% CI) | Standard estimates | Altered Train a Program and Minecraft Bot estimates | All TH2 estimates above a 0.5 score made 4 times longer |
|---|---|---|---|
| GPT-6 Astra | 40 h (13–2645) | 29 h (14–946) | 32 h (13–120) |
| Opus 5.5 | 39 h (12–1253) | 30 h (13–450) | 29 h (12–111) |
The shift is similar enough across models that it changes the TH2 verifiable trend minimally.
Data Copy this chart's data
Figure 27: Time horizon trends by task set under alternative time estimates

Each panel refits the same runs after changing only the estimated human times of score thresholds. TH2 does not contain many short tasks, so to constrain its logistic fits we add short (<10m) TH1.1 tasks to TH2 in this figure. We also show trends on MirrorCode, another verifiable task distribution. Trends are fit to frontier models. Doubling time CIs are calculated by bootstrapping over models, families, and runs. Model CIs are bootstrapped over families and runs.
80% Time Horizon Copy link to this section
In this figure, we show 80% time horizons, instead of 50% time horizons.
Data Copy this chart's data
Figure 28: When separated, 80% time horizon doubling times on TH2 (verifiable) appear shorter than on TH1.1.

We also show trends on MirrorCode, another verifiable task distribution. Trends are fit to frontier models. Doubling time CIs are calculated by bootstrapping over models, families, and runs. Model CIs are bootstrapped over families and runs.
C MCI Modelling Methodology Copy link to this section
Our goal with METR Capability Index (MCI) is to estimate AI capabilities and task difficulty along a single axis, in a similar spirit to the ECI and AECI. For these purposes, Item Response Theory has a few benefits, including allowing us to evaluate agents only on a subset of questions, accounting for differences in how difficult and discriminatory tasks are, and quantifying uncertainty in our capability estimates.
Data Copy link to this section
First, we need to form a dataset of how an agent performed on various tasks. This is complicated somewhat by the fact that we have lots of closely related tasks, and sometimes have multiple agent attempts on a given task. Thus, we conceptualize our data in a hierarchical structure, where each hierarchy of data serves a different purpose in our IRT model:
- families: these are related groups of tasks. For instance, we have the AZsPCs family where the tasks are the individual competitions. Our data has 113 families.
- tasks: these are individual activities with goals the agents might pursue. Our data has 333 66 tasks, meaning an average of 2.9 tasks per family.
- runs: We often evaluate an agent on a task multiple times. A run is an independent instance of an agent performing a task. Our data has 38,180 scored run-level observations, meaning an average of ~4.7 observations per evaluated agent-task pair.
The fundamental unit of analysis is a task, but we use runs to help more precisely estimate how an agent might do on a given task, and we use families to downweight tasks that are very similar.
Basic IRT Model Copy link to this section
For each of our tasks, we define a scale in [0,1], where 0 corresponds to failure and 1 corresponds to full score 67, and fit a two parameter logistic for the score an agent (m) will get on a task (t):
[p_{mt}=\sigma (\alpha_{t}(\theta_{m}-\beta_{t})),\quad \sigma (x)=\frac{1}{1+e^{-x}}]
where (p_{mt}) is the expected score agent (m) will get on task (t), (\alpha_{t}) is the task discrimination, (\beta_{t}) is the task difficulty, and (\theta_{m}) is the agent ability.
A difference of 0 between task difficulty, (\beta_{t}), and agent ability, (\theta_{m}), predicts that the agent will score 0.5 on that task. The slope parameter (\alpha_{t}) corresponds to how well a given task can distinguish between two agent abilities, in that a one unit increase in agent ability (\theta_{m}) predicts an (\alpha_{t}) unit increase in the log score odds.
Task Dispersion Copy link to this section
Since our task suite is very diverse, we want to downweight ‘noisy’ tasks where our observed data is not well fit by our model. So, we fit a noise parameter for each task, called the dispersion ((\varphi_{t})), which is a function of run-level deviance ((d_{mtr})):
[\varphi_{t}=\frac{\sum_{m,r}d_{mtr}}{N_{t}-2},\quad d_{mtr}=2[y_{mtr}\log \frac{y_{mtr}}{p_{mt}}+(1-y_{mtr})\log \frac{1-y_{mtr}}{1-p_{mt}}]]
where (y_{mtr}) is the observed score for a run of an agent on a task. Mathematically, dispersion is approximately the average of the difference in log-likelihood between the fitted curve and the observed curve. Intuitively, dispersion measures how much individual run outcomes depart from what the fitted curve predicts. Larger departures mean we treat the task as noisier and give it less weight.
Fitting the IRT Model Copy link to this section
To fit our two-parameter logistic, we maximize the fractional-logit log-likelihood. To reflect certain features of our data, we introduce a few adjustments to the fit:
- Downweighting of noisy tasks. We multiply by (1/\varphi_{t}), which is the inverse of our task dispersion parameter, to downweight noisy tasks.
- Partial downweighting on run count. We want more runs for a task to reflect increased confidence for a task, while not completely discounting tasks with only a few runs. So, we apply (w_{mt}=1/\sqrt{n_{mt}}), where (n_{mt}) is the number of scored runs for an evaluated agent-task pair.
- Partial downweighting on family size. Since families often include similar tasks and we want our model to be robust to adding many tasks within a given family, we add family-level down-weighting, (f_{t}=1/\sqrt{q_{t}}), where (q_{t}) is the number of fitted tasks in task (t)'s family.
Further, to help with model fitting, we add priors on our task discrimination and difficulties. In particular, our base priors are:
[\log \alpha_{t}\mid \mu \sim N(\mu ,3^{2}),\quad \beta_{t}\sim N(0,{10}^{2})]
We scale each task's prior penalties by (f_{t}c_{t}), where (c_{t}=\frac{\sum_{m}\sqrt{n_{mt}}}{\sum_{m}n_{mt}}), the average run weight. We fit (\mu) as the weighted mean log discrimination, (\frac{\sum_{t}f_{t}c_{t}\log \alpha_{t}}{\sum_{t}f_{t}c_{t}}).
Our full objective thus becomes:
[\max_{\theta ,\alpha ,\beta }\quad \sum_{t}\frac{f_{t}}{\varphi_{t}}\sum_{m,r}w_{mt}[y_{mtr}\log p_{mt}+(1-y_{mtr})\log (1-p_{mt})]-\sum_{t}f_{t}c_{t}[\frac{(\log \alpha_{t}-\mu )^{2}}{2\cdot 3^{2}}+\frac{\beta_{t}^{2}}{2\cdot {10}^{2}}]]
The dispersion (\varphi_{t}) is not part of this maximization; we use double GLM, which allows parameters to have their own noise parameter. In particular, we alternate between fitting with (\varphi_{t}) held fixed and re-estimating (\varphi_{t}) from the fitted curves, until all the parameters stop changing. The dispersion (\varphi_{t}) is estimated from unweighted run deviances; neither family nor run weights enter its update.
We establish a scale for the ability and task difficulty axis by setting two anchor models: GPT-4o at -1, and Opus 4.6 at 0, and rescale the axis so that GPT-4o is at 0 and a one-unit change in ability is set equal to 633, which is the number of days between each model's release.
Confidence Intervals Copy link to this section
To produce our confidence intervals, we do a hierarchical bootstrap over families and then runs. Each sampled copy of a family independently resamples within each model–task pair. We resample whole task families because related tasks may provide overlapping evidence about model capability. Resampling runs captures variation between repeated evaluations.
Differences from the ECI Copy link to this section
Our analysis is similar in spirit to Epoch’s ECI, but we do have a few notable differences, including:
| Source of difference | How this is reflected in our model |
|---|---|
| We run a task-based analysis, while the ECI runs a benchmark-based analysis. | We’d expect our tasks to vary more in their reliability than benchmarks, since benchmarks aggregate over many questions or tasks. Thus, we downweight ‘noisy’ tasks where our observed data is not well fit by our model using our dispersion parameter. |
| We have run-level data, and each run enters the fit as its own observation. The ECI fits one score per model-benchmark pair. | We want the fit to account for how much evidence supports each observed average. Thus, we make it so that additional runs on a task count for more data. In particular, a model-task pair’s weight grows with its number of runs, and run-to-run variation contributes to the task’s dispersion parameter and bootstrap. |
| We have the structure of tasks being in families, whereas the ECI does not. | We downweight tasks according to the family size, so that certain families don’t dominate the analysis. We estimate uncertainty using a hierarchical bootstrap over task families and runs, while the ECI reports posterior credible intervals. |
D Limitations of IRT Copy link to this section
IRT provides a useful way to summarize model performance, but the resulting ability estimates and trends should be interpreted with several caveats:
- The ability axis reflects models’ patterns of strengths and weaknesses across tasks, rather than those of humans. In other words, this figure is not interchangeable with the original time horizons graphs – a linear trend on this axis may not necessarily correspond to linear growth in the human time required to complete tasks that the models can solve.
- We think IRT methods will likely be a lagging indicator of acceleration. When the most capable models reliably succeed on tasks that distinguish weaker models, there may be few informative comparisons available to estimate the size of their ability advantage. Without regularization, this can lead to wide confidence intervals on their estimated abilities, making acceleration difficult to distinguish from statistical uncertainty. As a result, a substantial capability gain may only become clear retrospectively, once additional models or harder tasks provide more informative comparisons.
- These trends do not account for internal models. It’s possible that internal models are far beyond the current frontier.
- Results can be sensitive to the IRT modeling decisions. To account for this, we conduct a variety of robustness tests, and generally find similar trends (see IRT robustness checks in Appendix E: MCI Robustness Checks). We also purposefully do not place any priors on IRT ability, which can artificially lower confidence intervals.
- Changes in task composition can also affect the trend if different capabilities improve at different rates. The distribution of tasks here also may not reflect the most important capabilities necessary for frontier lab uplift, or for AI risk scenarios. We do not find any multi-dimensionality in performance (see IRT robustness checks in Appendix E: MCI Robustness Checks). However, these checks cannot establish that our task suite captures all the capabilities relevant to AI R&D or AI risk.
- Identifying acceleration is a product of good task-design. One consequence of moving away from human baselines is that, for some tasks, we no longer know for sure whether the highest possible scores are achievable. This makes it harder to measure progress, since models may approach an unknown performance ceiling and appear to improve more slowly, even when their scores are well below the line the task developer sets as the best possible performance. We believe that this hasn’t affected our tasks significantly thus far, but may be a bigger problem going forward.
E MCI Robustness Checks Copy link to this section
In this section, we assess our IRT model’s fit, test how sensitive our findings are to modelling choices, and examine what our ability estimates capture.
How well does the model describe the data?
- Examine residual plots for abnormal patterns.
- Check whether the data support a single ability dimension.
How sensitive are our findings to modelling choices?
- Vary the strength of our priors.
- Use a least squares objective instead of maximising a fractional-logit log-likelihood.
- Change the choice of anchors.
- Remove the down-weighting.
- Change the link function.
What do our ability estimates capture?
- Test whether our analysis would detect acceleration if it occurred on our task suite.
- Compare our ability estimates with ECI estimates.
- Fit a linked-linear line to our IRT ability estimates (see Appendix H).
In general, we find that our current assumptions are reasonable and that most modelling choices do not result in large changes to our ability estimates.
Residuals Copy link to this section
First, we inspect our residuals, which additionally serves as a check for our choice of the logit link function. The following plots compare predicted score and actual score, to diagnose whether the logit link function is a good choice. We find that the link function does fit our data well, both for continuously scored tasks and for binary tasks.
Data Copy this chart's data
Figure 29: IRT residuals on continuously scored tasks

Each point is a model–task pair. For model-task pairs with several runs, the y-value is the average score over that pair’s runs.
Data Copy this chart's data
Figure 30: IRT residuals on pass/fail tasks

Each point is a model–task pair. For model-task pairs with several runs, the y-value is the fraction of runs that succeeded over that pair’s runs.
Multidimensionality Check Copy link to this section
In our main analysis, we assume that ability is unidimensional. In this robustness check, we try fitting the model with more than one dimension to see if this assumption is reasonable. We employ the standard multidimensional 2PL model:
[p_{mt}=\sigma (\alpha_{t}(\theta_{m}-\beta_{t})+\sum_{k=1}^{K} {}\lambda_{tk}\psi_{mk}),\quad \sigma (x)=\frac{1}{1+e^{-x}}]
Where (K) is the number of extra dimensions, (\psi_{mk}) is model (m)’s score on extra ability dimension (k), and (\lambda_{tk}) is how much task (t) depends on this extra ability dimension beyond (\theta_{m}).
We find that ability is unidimensional in our data. First, almost all differences between models lie along one ability axis. Second, the model with one-dimensional ability fits better according to the BIC.
Table 2. BIC results across extra dimensions
| Extra dimensions | Extra parameters | BIC difference (bigger is worse) |
|---|---|---|
| +1 | +366 | +2,268 |
| +2 | +731 | +4,840 |
We can visually compare between our two-dimensional ability model, and we find that the difference between models lies primarily along our main (\theta_{m}) axis.
Data Copy this chart's data
Figure 31: Almost all differences between models lie along one ability

Strength of penalties Copy link to this section
We include prior penalties on (\alpha_{t}) and (\beta_{t}):
[\log \alpha_{t}\mid \mu \sim N(\mu ,3^{2}),\quad \beta_{t}\sim N(0,{10}^{2})]
The (\alpha_{t}) penalty punishes tasks where the log discrimination is very far from the average log discrimination in the suite; (\beta_{t}) penalty punishes tasks whose difficulty is far from the level at which Opus 4.6 would have a 50% chance of success. Both priors are pretty weak, but especially the task difficulty prior. The log discrimination prior is necessarily somewhat stronger to help our CI bootstraps converge.
We find that changing the strength of the prior penalties changes our results very little.
Data Copy this chart's data
Figure 32: Robustness to difficulty prior SD (baseline 10)

Data Copy this chart's data
Figure 33: Robustness to log discrimination prior SD (baseline 3)

Least Squares Copy link to this section
In our main analysis, to fit our two-parameter logistic, we maximize the fractional-logit log-likelihood. We do this since fractional-logit gives greater weight to a given prediction error near 0 or 1 compared to least squares. This fits our noise structure better, because we believe that small score changes near 0 or 1 can represent meaningful capabilities differences that least squares gives little weight to.
In this robustness check, however, we try minimizing the least squares objective function. This involves a task-dispersion that looks like:
[\varphi_{t}=\frac{\sum_{m,r}d_{mtr}}{N_{t}-2},\quad d_{mtr}=2(y_{mtr}-p_{mt})^{2}]
And a full objective function that looks like:
[\max_{\theta ,\alpha ,\beta }\quad -\sum_{t}\frac{f_{t}}{\varphi_{t}}\sum_{m,r}w_{mt}(y_{mtr}-p_{mt})^{2}-\sum_{t}f_{t}c_{t}[\frac{(\log \alpha_{t}-\mu )^{2}}{2\cdot 3^{2}}+\frac{\beta_{t}^{2}}{2\cdot {10}^{2}}]]
We find that changing to a least squares objective function changes our point estimates very little. It does slightly increase the ability scores for the pre-2025 models, and significantly widens their confidence intervals.
Data Copy this chart's data
Figure 34: Robustness to different objective functions

Sensitivity to Anchor Copy link to this section
In our main analysis, we choose GPT-4o and Opus 4.6 to anchor our ability axis at -1 and 0, respectively (0 and 633 on the MCI scale the charts show). We use anchor models because our axes would be under-determined otherwise; different combinations of ability, task difficulty, and task discrimination would give the same predicted score. We choose these two anchor models in particular to get coverage across our suite: both of them have been evaluated across many of our TH1, TH1.1, and TH2 tasks.
In this robustness check, we see whether choosing two different anchor models would change our results.
We find that the choice of anchor models changes our point estimates very little. However, as is expected, the confidence intervals for models farther away from the anchor models generally increase; this is especially true if the two anchor models chosen do not have good coverage across our suite (e.g., Opus 4.5 and GPT-5.4 haven’t been evaluated much on the TH1 and TH1.1 suite, so using them as anchors results in larger confidence intervals for models that have only been evaluated on those suites).
Data Copy this chart's data
Figure 35: Robustness to anchors

Remove downweighting Copy link to this section
In our task, we downweight noisy tasks with our dispersion parameter, partially downweight tasks by family membership which proxies for task similarity, and partially upweight tasks depending on how many runs it has.
Each of these weighting schemes is meant to handle our particular data structure. For instance, if we don't downweight tasks by family membership, certain categories of tasks (e.g., Slay the Spire and the Al Zimmermann contests) can contribute a disproportionate share of information constraining ability estimates. It happens to be that these families supply information that agrees with everything else, but weighting is a principled way to limit the dependence on those particular families. Note that there’s a sliding scale here; we can treat individual tasks as observations, which is like "no weighting" has occurred, or we can treat whole family units as observations, which is conceptually equivalent to weighting each task by 1/(number of tasks in the family). We instead take a middle-of-the-road approach, where we weight each task by 1/sqrt(number of tasks in the family). Similar logic applies to the weighting based on the number of runs. We think these are all reasonable choices given our data structure, but do complicate our model.
In this robustness check, we see how our results would change if all such weighting were removed.
...[1008 lines truncated]
Figure (fig36): Robustness to removing weighting

Figure (fig37): Robustness to link function

Figure (fig38): Robustness to injected acceleration

Figure (fig39): MCI vs ECI

Figure (fig40): Splitting tasks by Lifecycle Cost, controlled for task time
