Is there any reliable benchmark data showing actual developer productivity gains from using GitHub Copilot versus cheaper open source alternatives?
Download run
Fan-out diagram
The searches the AI actually ran for this prompt.
SEED QUERY
Is there any reliable benchmark data showing actual developer productivity gains from using GitHub Copilot versus cheaper open source alternatives?
4 real searches · 1 real-search pass
GitHub Copilot productivity randomized controlled study 55% faster developers experiment 2023Google ↗
OBSERVEDConsequence
AI coding assistant productivity field study Accenture GitHub Copilot 2024 developer productivity controlled studyGoogle ↗
OBSERVEDReason
METR 2025 randomized trial experienced open source developers AI coding tools 19% slowerGoogle ↗
OBSERVEDConsequence
GitHub Copilot vs open source coding assistants benchmark productivity Aider Continue Cline models studyGoogle ↗
OBSERVEDComparison
Badge and edge weight = share of passes that produced the query. Hover a query for its intent and routing format.
Query classes
Google's semantic query class for every sub-query: what kind of answer each search is after, simulated vs real.
ChatGPT real searches (4)
Classes come from the query-classifier (a local Laya model). 3 of 4 calls are low-confidence (under 50%) — hover a query in the table to see its confidence.
Searches and sources
ChatGPT4 real searches · gpt-6
GitHub Copilot productivity randomized controlled study 55% faster developers experiment 2023 Google ↗
AI coding assistant productivity field study Accenture GitHub Copilot 2024 developer productivity controlled study Google ↗
METR 2025 randomized trial experienced open source developers AI coding tools 19% slower Google ↗
GitHub Copilot vs open source coding assistants benchmark productivity Aider Continue Cline models study Google ↗
- The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers · doi.org
- The Impact of AI on Developer Productivity: Evidence from GitHub Copilot · arxiv.org
- Research: quantifying GitHub Copilot’s impact on developer productivity and happiness · github.blog
- metr.org · metr.org
- We are Changing our Developer Productivity Experiment Design · metr.org
- Official website · aider.chat
- Official website · continue.dev
- Official website · cline.bot
Queries
4 queries · click a column to sort
| Query / type | Search | Intent | Query class | Routing format | Cluster | Freq ↓ | Weight |
|---|---|---|---|---|---|---|---|
GitHub Copilot productivity randomized controlled study 55% faster developers experiment 2023 OBSERVED | Google ↗ AI Mode ↗ | — | Consequence | — | — | 1x of 1 | |
AI coding assistant productivity field study Accenture GitHub Copilot 2024 developer productivity controlled study OBSERVED | Google ↗ AI Mode ↗ | — | Reason | — | — | 1x of 1 | |
METR 2025 randomized trial experienced open source developers AI coding tools 19% slower OBSERVED | Google ↗ AI Mode ↗ | — | Consequence | — | — | 1x of 1 | |
GitHub Copilot vs open source coding assistants benchmark productivity Aider Continue Cline models study OBSERVED | Google ↗ AI Mode ↗ | — | Comparison | — | — | 1x of 1 |