A single "AI maturity score" hides more than it shows. A team can raise its adoption number while quality falls, or cut cycle time by skipping review. This chapter defines a balanced metric tree: five branches that are read together, measured per use case and per role, never as one organizational number. All numbers on this page are illustrative.
1. The metric tree
Each branch answers a different question. A gain on one branch counts only if the others hold.
| Branch | Question it answers | Fails if read alone because |
|---|---|---|
| Adoption | Are the right people using approved tools, and how deeply? | Usage can rise while value and quality fall |
| Value | Is the work faster, cheaper or better, after review and rework? | Time saved can come from skipped review |
| Quality | Is the output correct, and does it stay correct after release? | Quality can hold while nobody uses the tool |
| Risk | Are controls working, and are problems found and fixed? | Low incident counts can mean low detection |
| Sustainability | Will this keep working when people, models and assets change? | A healthy pilot can depend on one person |
Read every metric with three things. A denominator (per what), a baseline (compared to what), and a slice (for which role, use case and risk tier). A metric without these three is a talking point, not a measure.
2. Adoption
Adoption metrics show reach and depth. They are the easiest to collect and the easiest to game, so they are never reported without the value and quality branches next to them.
| Metric | Definition with denominator | Data source | Owner | Anti-gaming note |
|---|---|---|---|---|
| Eligible-user adoption | Users active on an approved tool in the period ÷ users who have finished foundation training and have access | Tool usage logs; training records | Enablement lead | Not total seats. A user who opens the tool once is not "active"; define active as use on a real task in at least 2 of 4 weeks (illustrative) |
| Stage reached per role | Teams at each ladder stage for a given role ÷ teams with that role | Stage exit evidence in the evidence system of record | Enablement lead with the team lead | A stage counts only with its exit evidence. Never set stage targets; a high-risk use case may correctly stay at a lower stage |
| Use-case coverage | Intake-approved use cases in active use ÷ intake-approved use cases | AI inventory; usage logs | Enablement lead | Approved but unused is a finding to explain, not a number to hide |
| Learning path completion | People who finished a level with a demonstrated artifact ÷ people enrolled at that level | Learning records; lab artifacts | Enablement lead | Completion without an artifact does not count |
3. Value
Value is measured on the whole task, not on the AI step. Net time saved includes the time spent reviewing, correcting and redoing AI output.
| Metric | Definition with denominator | Data source | Owner | Anti-gaming note |
|---|---|---|---|---|
| Net time saved | (Baseline time per task − current time per task including review and rework) × tasks in the period | Time samples; work item timestamps; reviewer logs | Team lead | Self-reported savings overstate. Use sampled measurement against a written baseline, and include the reviewer's time |
| Task success | AI-assisted tasks accepted without major rework ÷ AI-assisted tasks attempted | Pull request (PR) and ticket outcomes; reviewer decisions | Team lead | Define "major rework" before measuring. Do not drop failed attempts from the denominator |
| Override or rejection rate | AI outputs a reviewer changed materially or rejected ÷ AI outputs reviewed | Review records; approval gate logs | Team lead with QA (quality assurance) | A rate near zero is a warning, not a success: it can mean reviewers stopped checking (automation bias) |
| Cycle time | Median time from work started to change released, per work item type | Delivery tool timestamps; change records | Engineering lead | Read only with change-failure rate and escaped defects. Faster is not better if failures rise |
| Benefit realized vs expected | Measured benefit at review ÷ benefit stated in the leader decision brief | Leader decision brief; scorecard | Business owner | Compare with the original brief, not a revised one. Report shortfalls; they are how estimates improve |
4. Quality
Quality metrics show whether AI-assisted work holds up after release. Several are DORA metrics (DevOps Research and Assessment): cycle time, change-failure rate and time to restore.
| Metric | Definition with denominator | Data source | Owner | Anti-gaming note |
|---|---|---|---|---|
| Change-failure rate | Releases that caused a failure needing remediation ÷ releases | Change records; incident records | Release manager | Do not reclassify failures as "known issues". Slice by AI-assisted and not AI-assisted changes |
| Rework rate | Changes reopened or reverted within 30 days (illustrative) ÷ changes merged | Version control; ticket history | Engineering lead | Count reverts and follow-up fixes, not only reopened tickets |
| Escaped severe defects | Severity 1 and 2 defects found after release ÷ releases | Defect tracker; incident records | QA lead | Severity must be set by a rule, not by the team that shipped the change |
| Evaluation coverage | Delegated tasks with an evaluation set that meets its threshold ÷ delegated tasks in use | Evaluation reports; AI inventory | QA lead | An evaluation set with no negative test cases does not count as coverage |
| Regressions by slice | Evaluation cases that passed before and fail now, per risk slice, after a model, prompt or tool change | Evaluation reports | QA lead with AI risk | Report every slice separately. A good average can hide a failed edge-case slice |
5. Risk
Risk metrics show whether controls work. They need a reporting-culture caveat: when detection improves, counts often rise first.
| Metric | Definition with denominator | Data source | Owner | Anti-gaming note |
|---|---|---|---|---|
| Near misses | Reported near misses in the period, per 100 active users | Near-miss reports; incident system | Second-line risk | A rising count early on usually means people trust the reporting path. Never set a target to reduce it |
| Policy findings | Secret, data-classification and banned-pattern findings from hooks and CI (continuous integration) ÷ PRs scanned | Hook audit events; CI scan results | Security | Separate true from false positives; a hook that cries wolf gets bypassed |
| Exception age | Median and oldest age of open exceptions; exceptions past expiry ÷ open exceptions | Exception register | Control owner | An exception renewed without review is still past expiry |
| Trace completeness | Delegated or agentic runs with a complete trace (inputs, tools, outputs, approver) ÷ runs | Trace store; approval logs | Platform owner | Sample traces by hand each month; a trace that exists but is empty does not count |
| Customer harm and complaints | Complaints or harm events linked to AI-assisted changes ÷ customer-facing releases | Complaint system; incident records | Business owner with compliance | Link by change record, not by guesswork after the fact |
Reporting culture. In the first quarter of a program, near misses and policy findings often go up. That is usually the sign of better detection and safer reporting, not worse behavior. Compare like with like: same detection rules, same reporting channel, over at least two periods.
6. Sustainability
Sustainability metrics show whether the gains survive staff changes, model changes and asset ageing.
| Metric | Definition with denominator | Data source | Owner | Anti-gaming note |
|---|---|---|---|---|
| Reuse rate | Reuse or extend decisions ÷ all reuse decision records for changes that touched a cataloged capability | Reuse decision records; asset catalog | Architecture | Create is sometimes right. A justified new asset with a recorded rationale is a good outcome. Never target 100% reuse |
| Asset health | Cataloged assets with a named owner, supported status and review in date ÷ cataloged assets | Asset catalog | Architecture | A stale entry that the hook still recommends is worse than no entry |
| Deprecation compliance | Uses of deprecated assets past their removal date ÷ uses of deprecated assets | Dependency scans; catalog | Architecture | Track per asset, not as one percentage |
| Model change response time | Days from a provider or model change notice to re-evaluation complete | Change notices; evaluation reports | AI risk with platform owner | Count from the notice, not from when the team noticed a problem |
| Contribution rate | Teams that contributed an asset, pattern or catalog fix in the period ÷ teams at Stage 2 or above | Catalog history | Enablement lead | A contribution counts once reviewed and accepted |
7. Baselines and comparison periods
A number means nothing without a starting point. Set the baseline before the coaching cycle begins.
| Rule | How |
|---|---|
| Baseline first | Measure the task for two to four weeks (illustrative) before AI is introduced, with the same definitions |
| Same denominator | Use the same population and work types in both periods; new joiners and new work types go in separately |
| Two periods minimum | Compare at least two periods after the change; the first period after a launch is usually noisy |
| Comparison group | Where possible, compare with a similar team or task that has not changed yet |
| Slice, then sum | Report by role, use case and risk tier first; any total comes after |
| Freeze definitions | Changing a definition restarts the baseline; record the change and the date |
8. A filled scorecard
The scorecard below is for the worked-example team in chapter 8, two releases after Stage 3 test generation began. Every number is illustrative.
| Branch | Metric | Baseline | Current | Read |
|---|---|---|---|---|
| Adoption | Eligible-user adoption | 9 of 20 | 16 of 20 | Healthy; four users not yet through foundation |
| Adoption | Stage reached | QA 2, developer 2, product owner 1 | QA 3, developer 3, product owner 1 | Product owner staying at Stage 1 is correct for this work |
| Value | Net time saved per change, incl. review | n/a | 3.5 hours | Review takes 1.5 of the 5 hours saved |
| Value | Override or rejection rate on generated tests | n/a | 18% | Expected; reviewers are checking |
| Value | Cycle time, display changes | 11 days | 8 days | Read with quality below |
| Quality | Change-failure rate | 12% | 8% | Improved; small sample |
| Quality | Escaped severe date defects | 3 per quarter | 0 in two releases | Too early to claim; keep watching |
| Quality | Evaluation coverage, temporal edge-case slice | 0 of 4 tasks | 4 of 4 tasks | Daylight saving time (DST) transition and missing-zone cases included |
| Risk | Near misses reported | 1 | 4 | Rise follows new reporting path; one was a proposed restricted-data flow, blocked |
| Risk | Oldest open exception | 140 days | 45 days | One exception expires next month; owner named |
| Risk | Trace completeness | n/a | 97% | Two runs missing approver field; fixed in config |
| Sustainability | Reuse rate, date and time changes | 2 of 10 | 8 of 10 | Two justified creates recorded with rationale |
| Sustainability | Asset health, shared-datetime | Owner unclear | Owner named, review in date | Catalog entry updated with the temporal-domain contract |