METR report claims AI is doubling Anthropic’s model improvement pace
An 11-week investigation by METR suggests Anthropic’s models are advancing twice as fast as last year, driven primarily by AI automation rather than compute scaling.
Key points
- METR reports Anthropic’s model improvements have accelerated roughly twofold compared to a year ago.
- Researchers attribute the majority of this acceleration to AI automation, not increased computing power.
- Anthropic redacted portions of the final report, limiting public visibility into specific risks and progress.
A report released Thursday by AI safety group METR indicates that Anthropic’s AI models are advancing at approximately double the rate seen a year ago. The investigation is distinct from a separate initiative that grants safety researchers access to systems akin to employees, established after significant hacking incidents occurred during the summer.A new report says Anthropic’s AI models are getting better twice as fast as last year because they are using AI to build AI.
How we got here
METR researchers examined Anthropic’s internal development processes to understand the drivers of recent performance gains. Historically, scaling up computing power has been a major factor in AI progress. However, the investigation sought to determine whether other factors, such as automated tooling, were contributing to the observed acceleration in model capabilities and benchmark scores.
What is new
Benchmark scores imply that Anthropic achieved approximately two years of historical progress within the past year. METR researchers expressed confidence that the acceleration was not merely the result of increasing the computing power used for model training.
What to watch
The results prompt inquiries into the speed at which technology might progress toward a scenario in which AI systems enhance themselves with minimal or zero human participation. This occurs as Anthropic prepares for an initial public offering, despite CEO Dario Amodei’s public appeals for AI labs to slow down development. The tension between rapid internal progress and external calls for caution remains a key focal point.
Availability and limits
As part of its deal with METR, Anthropic was allowed to remove sections from the final report. Anthropic’s extensive use of redaction rights has sparked uncertainty regarding the public’s understanding of the speed and dangers associated with advanced AI progress. The report provides a rare look inside leading AI labs, yet the omitted information is equally significant for those monitoring the industry.
Why it matters
- The report provides clear evidence that AI is accelerating its own development, raising questions about how quickly systems could advance with little human involvement.
- Anthropic’s models are improving roughly twice as quickly as a year ago, with researchers confident the acceleration is due to AI rather than just increased computing power.
- The findings come at a pivotal time as Anthropic prepares to go public while its CEO calls for labs to slow down AI development.
- Anthropic’s liberal use of redaction powers in the report raises questions about how much the public truly knows about the pace and risks of frontier AI development.
Topics: Anthropic, METR, AI safety, model development, automation
Updated
The strongest doubt
Counterpoint from the sources
Anthropic redacted portions of the report, which limits the ability to independently verify the extent of AI-driven acceleration and associated risks.
Did this article give you the evidence you needed?
Your choice stays on this device. No public tally.Questions readers ask
What drove the acceleration in Anthropic’s model improvements?
METR researchers attribute the majority of the acceleration to AI automation, rather than increased computing power.
Did Anthropic control what was published in the report?
Anthropic was allowed to redact parts of the final report and seems to have used that authority extensively.
How does the current pace compare to last year?
The report states that Anthropic’s models are advancing at approximately double the rate seen a year ago.
Connected stories
Reading receipt
- METR reports Anthropic’s model improvements have accelerated roughly twofold compared to a year ago.
- Researchers attribute the majority of this acceleration to AI automation, not increased computing power.
- Anthropic redacted portions of the final report, limiting public visibility into specific risks and progress.
Still open: How much of the specific technical methodology for AI-driven development was redacted from the report?
Still open: Will the acceleration continue at the same rate, or are there diminishing returns?