Two Reports Show AI Outrunning the Humans Who Check It

Anthropic's models are improving roughly twice as fast as they were a year ago, and the AI safety organization METR says the majority of that acceleration is itself due to AI. The finding landed in the same week a separate report put a price tag on what an autonomous model can produce in 30 hours: 948 working Linux kernel exploits.

Read together, the two documents describe one problem from opposite ends. Machine output is scaling. The human capacity to verify that output is not.

What METR found inside Anthropic

METR published its report Thursday after an 11-week investigation inside the company, evaluating Anthropic's AI research and development. The headline measurement: benchmark scores suggest "that Anthropic has made roughly two years of historical progress over the last year."

The researchers went out of their way to rule out the obvious explanation. They said they "are confident" the acceleration was not simply the result of scaling up training compute, historically the dominant driver of AI progress. Instead, as of October, the majority of the acceleration "is due to AI," with AI's contribution growing over time.

The employee-side number is the one engineers will argue about. Via informal Slack polls and interviews, staff using Anthropic's most capable internal model self-reported being around 12 times as productive as they would have been without AI assistance — roughly double what staff reported a few months earlier on an older model. Self-reported productivity is a soft instrument, and METR presented it as supporting evidence rather than as the primary measurement.

Two forward-looking lines in the report matter more than the multiplier. METR warned that "there are some signs that fully autonomous contributions will play a larger role soon." And it said it found no clear signs of diminishing returns, nor observable bottlenecks that could prevent "even faster acceleration."

Three finished models nobody outside the company has used

METR reported that Anthropic finished training three models in the past six months that it has not released. The report lists them by the codenames shared with METR — Model 2, Model 3 and Model 4 — completing training in July, August-September, and September-October respectively. METR also identified an unreleased model dating back to 2022, labeled Unreleased 2022. The researchers noted these were "the codenames shared with us; we understand they have different internal codenames."

That detail reframes the productivity figure: much of the measured acceleration comes from internal models the rest of the industry cannot benchmark against.

The redactions

Anthropic gave METR confidential research, model performance data and employee interviews, and METR said the company "shared more information than was initially scoped for this project, surpassing our expectations." It also had the contractual power to redact the public version, and used it.

"We submitted a ~100-page full report on our assessment to Anthropic. For the public version, Anthropic redacted a majority of the evidence we found most informative," METR wrote on X. Elsewhere it noted that "our pipeline of information was relatively narrow, and a small number of our requests for data were denied." The consequence, in the report's own words: "the public has to rely on METR's judgment much more than would be ideal, given the weight of the questions answered here."

Anthropic justified the redactions on the grounds that the information could benefit rivals; METR countered that "most of the redacted information was very high-level." Anthropic cofounder Sam McCandlish responded on X: "We redacted a lot, and I accept that this makes it harder for others to check METR's work." He argued that disclosing what drives the speedup "isn't the right thing to do, both because it's commercially sensitive and because it risks handing out a roadmap for others to go faster," and said the company plans to publish its own report on the pace of AI progress "in the coming weeks."

The other end: 948 exploits, $1.2M, 30 hours

The verification gap gets concrete in the second report. OpenAI published research titled "A thousand fewer paths to root," which, according to a Dev.to write-up by the developer posting as @DevKursat, describes an upcoming autonomous cyber model that discovered and weaponized 948 working Local Privilege Escalation exploits in the Linux kernel in 30 hours. The API compute cost alone came to $1,213,430.

The author's framing is an argument, not a finding, and should be read that way: he calls the run "an enterprise-scale denial of service attack on volunteer maintainers," and contrasts a lab with large GPU clusters against "a handful of underfunded, volunteer kernel maintainers triaging patches on weekends." His claim is that generating candidates took 30 hours while triaging, testing, verifying and backporting 948 kernel patches across LTS branches would take thousands of hours of human labor. He also argues that models can produce exploit proofs-of-concept easily but cannot guarantee a candidate fix won't introduce subtle concurrency or memory bugs in critical kernel paths — and that dumping raw vulnerability reports without funded upstream maintenance "is not responsible disclosure—it is offloading corporate R&D costs onto the backs of open-source volunteers."

Whether or not you accept that framing, the arithmetic underneath it is the same arithmetic in the METR report. Generation is now cheap and fast. Verification is still human, still serial, and still the bottleneck.

What to watch next

Three dated checkpoints are now on the calendar. Anthropic says its own report on the speed of AI progress is due "in the coming weeks," which will show whether the company's framing matches METR's. OpenAI said in September it is "making strong progress" toward an AI researcher that can "further progress on deep learning and alignment" by March 2028 — a target the METR findings make easier to take seriously. And Marius Hobbhahn, CEO of Apollo Research, told The Verge the report suggests a pace of AI-enabled R&D even faster than the AI 2027 scenario forecast, which placed a key self-improvement breakthrough in early 2027.

If you ship software, the practical version of this is narrower: assume the volume of machine-generated patches, reports and PRs aimed at your project goes up before your review capacity does. The METR report says there is no visible bottleneck on the generation side. There is a very visible one on yours.