In the study, published in July, the think tank Model Evaluation & Threat Research randomly assigned a group of experienced software developers to perform coding tasks with or without AI tools.
It was the most rigorous test to date of how AI would perform in the real world. Because coding is one of the skills that existing models have largely mastered, just about everyone involved expected AI to generate huge productivity gains.
In a pre-experiment survey of experts, the mean prediction was that AI would speed developers’ work by nearly 40 percent. Afterward, the study participants estimated that AI had made them 20 percent faster.
But when the METR team looked at the employees’ actual work output, they found that the developers had completed tasks 20 percent slower when using AI than when working without it. The researchers were stunned. “No one expected that outcome,” Nate Rush, one of the authors of the study, told me. “We didn’t even really consider a slowdown as a possibility.”
No individual experiment should be treated as the final word. But the METR study is, according to many AI experts, the best we have—and it helps make sense of an otherwise paradoxical moment for AI.
None of that means that AI can’t eventually be every bit as transformative as its biggest boosters claim it will be. But eventually could turn out to be a long time. This raises the possibility that we’re currently experiencing an AI bubble, in which investor excitement has gotten too far ahead of the technology’s near-term productivity benefits.
Read more | THE ATLANTIC