How to measure AI adoption in an engineering team

Adoption is easy to count and easy to misread. Seats assigned, prompts sent and pull requests opened all go up when a team starts using an AI coding tool. None of them tell you whether delivery got better.

The useful question is narrower: against this team’s own baseline, did cycle time, review, coverage or defects move, and in which direction?

Why a feeling is not a measurement

In a randomised trial, METR found experienced developers working in their own repositories were 19% slower with AI tools and believed they had been about 20% faster. The gap is the whole problem. If you ask the team whether the tool is helping, you will hear the belief. The baseline is how you check it.

Faros AI, across a large set of teams, found high AI adoption came with 98% more merged pull requests and a 91% rise in review time. Pull requests got larger. Bugs per developer ticked up. Lead time and change failure rate did not improve. More output, same delivery, more review load. A dashboard of “PRs merged” would have called that a win.

Vendor dashboards (including Copilot usage metrics) are useful for activation: who opened the tool, how often, on which surfaces. They are not a substitute for delivery outcomes. Treat usage as a leading signal and the four numbers below as the score.

Four numbers, taken twice

Pick numbers you can read from a system you already have. We use these four because they match the place the cost actually moved:

Number	What it is watching
PR cycle time	How long a change waits from open to merged
Review time	How long a person spends, or how long the change sits in review
Test coverage	Whether generated code is held to the same bar
Defect rate	Whether “almost right” code is escaping

Take them for one team, for a period long enough to include ordinary work, before you change the process. Change one thing: the review rule, the tool policy, repository context, or the guardrails in CI. Read the same numbers again. If they did not move, that is the result. Publish it internally anyway. A report that only contains the metrics that improved is how teams talk themselves into the METR gap.

What not to use as the score

• Lines of code, or pull requests opened. Both rise when generation gets cheaper.
• Prompt counts and “accepted suggestions”. Those measure the tool, not the team.
• A happiness score on its own. Useful beside the four numbers, misleading instead of them.
• A company-wide average that mixes a team on the tool with three teams that are not.

DORA’s 2025 report is the reason to keep a stability number in the set. AI adoption still tracks with less stable delivery. If you only watch speed, you will miss the cost.

A baseline you can defend

Write down, before the second reading:

• which team
• the dates
• the source of each number
• what you changed, and what you deliberately left alone

That note is what stops a later argument about whether the comparison was fair. It is also what a renewal conversation needs. The question at renewal is not whether people like the tool. It is whether the four numbers moved enough to justify the seats.

Where this sits in a rollout

1. Policy — which tools may see code.
2. Baseline — the four numbers on one team.
3. Review checklist and context engineering — process and repository changes.
4. Second reading — expand seats only if the numbers moved the way you agreed (rollout plan).

The readiness check will show which part of the system is thin. It will not give you these four numbers. Those have to come from your own pipeline. If you want that baseline taken with you, that is the AI-Readiness Audit.

FAQ

Can we use DORA metrics alone? Use whatever your team already trusts for stability and throughput. Keep at least one review-load number beside them; Faros-shaped outcomes hide if you only watch lead time.

How long is a fair baseline window? Long enough to include ordinary work for that team — usually weeks, not a single sprint demo. Match the length of the pilot that follows.

What if the numbers get worse? Report it. Stopping or redesigning the rollout after a clear readout is success. Quietly buying more seats after a worse review queue is how the cost moves downstream.