The clearest measured effect of AI on information technology work is in programming, and the studies point in different directions. A 2022 experiment found developers with an AI assistant finished one coding task 55.8 percent faster, a 2025 randomized trial found experienced open-source developers took 19 percent longer, and a pooled field experiment across three companies reported 26.08 percent more completed tasks. The spread comes from who was measured, which tasks they did, and when the tools were tested.
Key takeaways
- A 2022 experiment with 95 freelance developers measured a 55.8 percent shorter completion time on a single JavaScript task.
- A field experiment with 4,867 developers at three companies found 26.08 percent more completed tasks, with larger gains for less experienced developers.
- In a 2025 trial, 16 experienced open-source developers were 19 percent slower with AI tools, though they believed they were faster.
- METR's February 2026 update labeled that result out of date, and its new estimates had wide ranges that included no effect.
- The U.S. Bureau of Labor Statistics projects software developer employment to grow 10 percent from 2025 to 2035.
A single task in a controlled setting
Sida Peng and coauthors ran the first widely cited experiment from May 15 to June 20, 2022, just before GitHub Copilot became generally available. They recruited 95 professional developers through the freelance platform Upwork and asked them to write an HTTP server in JavaScript. Developers randomly given Copilot finished in an average of 71.17 minutes, against 160.89 minutes for the control group, a 55.8 percent reduction in completion time with a 95 percent confidence interval of 21 to 89 percent. The success rate was 7 percentage points higher for the treated group, but that difference was not statistically significant.
The task was small, well defined, and written for the study. Participants on average guessed a 35 percent gain, lower than the measured result, so intuition did not overstate the benefit here.
Real work at three companies
Zheyuan Cui, Mert Demirer, and colleagues pooled three randomized field experiments at Microsoft, Accenture, and an anonymous Fortune 100 company. The companies ran them as part of ordinary operations, and a random subset of developers received an AI code-completion tool. Across 4,867 developers, those with access completed 26.08 percent more tasks, with a standard error of 10.3 percent. Less experienced developers adopted the tool more often and gained more. Each experiment was noisy on its own, so the pooled figure carries most of the weight.
Experienced developers on their own projects
METR, a nonprofit research group, studied 16 experienced open-source developers working on 246 real issues in repositories averaging more than 22,000 stars and one million lines of code. Each issue was randomly assigned to allow or forbid AI tools, mainly Cursor Pro with Claude 3.5 and 3.7 Sonnet. Allowing AI increased completion time by 19 percent. Before the work, developers forecast a 24 percent speedup, and afterward they still believed AI had saved them about 20 percent. The authors examined 20 possible factors behind the slowdown and judged five as likely contributors.
The finding was specific to skilled people working in codebases they knew well, using early-2025 tools. It was never a measure of all software workers.
The 2026 update and why it is unsettled
On February 24, 2026, METR published a follow-up that began in August 2025 and covered 57 developers, 143 repositories, and more than 800 tasks, of which 10 developers returned from the first study. The estimate for returning developers was 18 percent less time, with a 95 percent interval from 38 percent less to 9 percent more. For new developers it was 4 percent less, with an interval from 15 percent less to 9 percent more. METR described the central estimates as a poor proxy for real impact, because many developers would not work without AI and between 30 and 50 percent said they withheld tasks they preferred to do with it. The group believes developers were likely more sped up in early 2026 than a year earlier, and states that its data are only very weak evidence of the size of that change.
The four measurements side by side
| Study | Setting | Result | Main limit |
|---|---|---|---|
| Peng et al., 2022 experiment | 95 freelance developers, one JavaScript task | 55.8 percent shorter time | Small, self-contained task |
| Cui et al., three field experiments | 4,867 developers at three companies | 26.08 percent more completed tasks | Noisy individually; task counts are not quality |
| METR, early 2025 | 16 experienced developers, 246 issues | 19 percent longer with AI | Tiny sample; METR calls it out of date |
| METR, late 2025 data | 57 developers, 800+ tasks | Intervals include no effect | Selection and withheld tasks |
What official statistics say about IT jobs
Productivity studies do not measure employment. The U.S. Bureau of Labor Statistics projects employment of software developers, quality assurance analysts, and testers to grow 10 percent from 2025 to 2035, much faster than the average for all occupations. The occupation group held about 1.9 million jobs in 2025, with a median annual wage of USD 134,040 in May 2025. The agency attributes the projected demand in part to continued software development for AI, the Internet of Things, robotics, and other automation. A projection is an estimate made at one date, and it does not settle how work inside those jobs will change.
Questions to ask about a claim that AI speeds up IT work
- Was the task chosen by researchers or drawn from the team's real backlog?
- Does the measure count time, completed tasks, or the quality of what shipped?
- How experienced were the participants, and did they know the codebase?
- Which tool version was tested, and when?
- Do the confidence intervals include zero?
FAQ
Does AI make developers faster?
It depends on the task and the person. Studies report large gains on a small defined task, moderate gains in company field data, and a slowdown for experienced developers on familiar projects in early 2025.
Why do developers misjudge their own speed?
In the 2025 METR trial, developers expected a 24 percent speedup and believed afterward that AI had helped by about 20 percent, while the measured result was a 19 percent slowdown. Self-reports and stopwatch results can diverge.
Is the 2025 slowdown result still valid?
METR marks it as out of date for current tools. Its 2026 update did not give a clear replacement figure.
Are IT jobs disappearing?
The Bureau of Labor Statistics projected 10 percent growth for software developers from 2025 to 2035. A projection is made at one date and does not describe current hiring at any one level of experience, so it should be read apart from productivity results.
Related reading
- How AI systems adapt to new information
- AI technology in education
- AI and robotics in society
- Where AI and art meet
More in AI Basics.
Last verified: October 2026
Sources
- The Impact of AI on Developer Productivity: Evidence from GitHub Copilot, Peng S, Kalliamvakou E, Cihon P, Demirer M, arXiv, February 2023
- The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers, Cui Z et al., Microsoft Research working paper, 2025
- Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, Becker J, Rush N, Barnes E, Rein D, METR, July 10, 2025
- METR uplift update, METR, February 24, 2026
- Software Developers, Quality Assurance Analysts, and Testers, Occupational Outlook Handbook, U.S. Bureau of Labor Statistics, 2025 data