AI-SDLC, AI agents in your delivery: what changes in your work

What runs at each stage

• Ticket: you write the task on your laptop or in Jira; cloud agents build and check clear tasks, you steer unclear or risky ones on your laptop; a person approves every product change, and risky ones also need the code owner.
• Plan: a short rules file, the context repository (glossary, runbooks, ADRs) and a map of which folder owns what.
• Build: many cloud agents, each task in its own sandbox; permissions keep merges and protected paths with people; a router picks the model per step, and when a model or a whole tool stops, the next on your list takes over.
• Review, in run order: a reviewer agent checks the implementer's work, a second model family cross-reviews, a cloud agent pre-reviews the pull request (you can run it on your laptop first), then a person approves.
• Merge: CI runs tests that must fail when the code is broken on purpose; a security-watch agent opens dependency fixes; gates check every pull request; a merge queue tests each change against the latest main, and a repair agent asks a person after 3 failed fixes.
• Release: your deploy pipeline releases after required checks and signatures; rollback is one step. One record links ticket, scope, approval and release.
• Run: a script checks error rate and CI failures against their normal range; outside it, an agent drafts a ticket with the evidence and a person decides.
• Measure: the five team numbers weekly, AI work split from other work; cost per change measured and kept under its cap; long sessions kept short.
• Hand over: short reports with a fact check; a named owner and a monthly review for each setup.

The problem each setup answers

• Who does what: unclear AI steps.
• Delivery process: vague tickets.
• Goal convergence: agent drift.
• Rules and context: ignored rules.
• Context repository: guesswork.
• PRD and ADR: forgotten decisions.
• Agent permissions: unattended agents.
• Model routing: one provider.
• Multi-agent: usage limits.
• Implementer and reviewer: self-approval.
• Cross-review: blind spots.
• Code Review: review queue.
• Testing: weak tests.
• Security: vulnerable dependency.
• Quality gates: flawed AI code.
• CI on main: red main.
• Tracing: untraced changes.
• Production monitoring: late incidents.
• Five numbers board: unproven gains.
• AI cost levers: rising AI bill.
• Token savings: long sessions.
• Kill verbosity: unreadable reports.
• Handover: setup decays after we leave.

Client case, anonymised (our measurement)

Test-automation SaaS company.

• More than half of its classified production incidents came from its own changes, and the share rose as output rose.
• Fix: the existing incident-cause field goes on the weekly board, so each change, agent-built or not, is measured against what comes back.

Client cases, anonymised (our measurement), and public reports

Test-automation SaaS company, ticket agent.

• Nobody knew which tickets the agent should take.
• Fix: hand it small fresh tickets the week they are written; people keep the rest.
• Fresh tickets closed in a median of 2 days, backlog tickets in 67 (fresh tickets are also smaller).

Enterprise device maker.

• Seven steps: ticket, design, build, check, pull request, review, merge.
• Three gates where a person decides: design, code and merge, which agent pull requests pass too.
• The author is never the reviewer.

Software platform company.

• The check moved from reading the diff to just before deploy (tests, evaluations, a watched feature-flag rollout).
• One person reads the AI review and CI results.
• Merges rose about 4.5x at this company (self-reported; its own post reports pull request cycle time falling from 21 h to 5.4 h).

Runs alone

• Build: an agent writes the code and its tests on its own branch, with no production keys.
• Second model checks: another model reads the change and flags the lines a person must read; it never fixes or approves.
• Build, tests, scans: build, types, lint, tests, code scan, packages and secrets (gate table). A failed gate goes back to the agent, up to 3 rounds; then the ticket returns to the developer.
• Release: your deploy pipeline releases after required checks and signatures; rollback is one step. The version is written on the ticket.

Always a person

• Ticket: the acceptance check that proves it is done. No check, no agent start.
• Review: AI pre-reviews every change; a person other than the requester approves every product change.
• Sign-off: risky changes (payments, login, personal data, infrastructure) also need the code owner. Low-risk changes skip it.
• Any change that weakens a test, a gate or a rule goes to the code owner.

One list shows every ticket waiting for a person, the step it stopped at, and why.

Who can override

• A failed check: only the code owner, with a written reason kept on the ticket.
• An agent: never.
• Anyone on the team can stop a release; rollback is one step.

Retry limit, risky paths and the 400-line cap per pull request (see Code Review) are our defaults, set with the pilot team in week 1 and tuned from your numbers.

Run: production monitoring

• Range check: after each release, a script compares the error rate and CI failures with their normal range.
• The agent drafts a ticket: when a number leaves its range, it writes the evidence (release, change, graph) into a new ticket. It never fixes or rolls back on its own.
• A person decides: roll back, fix or close. A fix takes this same path from the ticket.
• Each incident records its cause, so changes from your own team, agent-built or not, show on the weekly board.

Client case, anonymised (our measurement): test-automation SaaS company. More than half of its classified production incidents came from its own changes, and the share rose as output rose. Fix: the existing incident-cause field went on the weekly board.

Default and alternatives

Default: three files in the repository (spec, plan, task list, from GitHub Spec Kit: spec.md, plan.md, tasks.md), run by cloud agents (Claude Code), gated by your CI and the official code-review plugin. Spec Kit supplies the commands and files, not a runner; we configure the implementer and reviewer agents.

Alternatives and when each fits: Choose the delivery process.

How it works

• /speckit.specify reads the approved PRD and writes spec.md with its acceptance checks.
• The plan command reads the accepted ADRs and writes plan.md.
• /speckit.tasks writes tasks.md; the task template set up in Implementation gives one task per acceptance check.
• Before writing the plan, the agent checks the code to avoid asking for what exists; one plan once asked for tests already written.

• You write the task on your laptop or in Jira; a cloud agent picks it up, and each task runs in its own sandbox.
• A reviewer agent checks the diff against spec.md and tasks.md; problems go back to the implementer.
• When the work departs from plan.md, the plan is updated in the same commit.

The pull request then meets your CI and the code-review plugin. Branch rules keep the merge with a person.

Review order

1. Reviewer agent: checks the diff against the spec; the implementer answers each problem.
2. Second model: another model family reads the change without the author's reasoning.
3. Pull-request pre-review: risk labels decide who reads which lines.
4. Person: approves every product change; nobody approves their own.

An AI check blocks only after 90% precision and 60% recall on 30 past changes, labelled by two people (agreement measured) and not used to tune it (held out); until then it warns. Agent pull requests get human-only review about 8% of the time against about 25% for human ones (arXiv:2605.02273), so the person stays last.

Where it fits in the AI-SDLC

• Choose the delivery process (Discovery and analysis) picks it from your maturity scores, stack and tickets; small changes take a light path, a one-page spec.
• Set up the delivery process (Implementation) gives the pilot team the templates, the two agents and the gate.

When it does not help

• Thoughtworks and Martin Fowler's site find the full artifact set heavy for small work.
• No published study shows that a spec step improves delivery outcomes, so the pilot measures the five delivery numbers before and after.

How it works

Three files hold the work: the task board (what is left, an acceptance check per task), the decisions (your rulings), the work log (what happened).

Only a sentence you wrote can raise a task to the top priority; the agent cannot promote its own work.

An automatic check validates the board; a timer re-reads the files so a long session stays on them.

Where it fits in the AI-SDLC

Implementation. The pilot team's agents work the roadmap tasks from Discovery and analysis, and each task's acceptance check becomes a number checked in Measure and hand over.

Result

No published number yet.

When it does not help

A single-step task, or work where a runnable test already decides what done means.

Client cases, anonymised (our measurement), and public reports

Social network company, data pipelines.

• Agents lacked the team's unwritten knowledge of a four-repository, three-language pipeline.
• Each investigation took about two days.
• Fix: agents read 4,100+ files and wrote 59 reference files, graded by separate critic agents.
• Investigations fell to about 30 minutes; tool calls per task fell by 40%.

Ride-hailing and delivery platform. With a map of data and service metadata (not of code) the same question to an agent, finding the right analytics table, took 38 seconds and was right; without it, 20 minutes and wrong (one practitioner's report).

Enterprise device maker. One command writes the project's agent rules file; company, project and personal rules load in that order.

• Agents reach company systems only through approved connectors; unapproved tools stay blocked, so security knows what agents can read.

When a rules file pays off

• Generic context file
  • Over 20% more model cost, no general gain in task success
  Gloaguen et al.
• Map written for one codebase
  • Reverse-engineering estimate cut from 6 weeks to 2, for about 10,000 lines of COBOL
  Thoughtworks

So a rules file helps only when it is short and specific to your code.

How we keep it that way

• Size
  • A CI check measures it: Codex reads only the first 32 KiB of an AGENTS.md and drops the rest silently
  Open Mercato, open-source project
• Each rule
  • Backed by a CI check
  • Dropped if it does not change the agent's work
  • A mistake the agent repeats becomes a rule, with its check
• Rules file, agent-step scripts or model change
  • The checks rerun on past tasks with known good results
  • A lower pass rate stops the change

Client case, anonymised (our measurement)

Social network company, data pipelines.

• Agents lacked the team's unwritten knowledge of a pipeline of 4,100+ files; they “weren't making useful edits”.
• Fix: 59 context files that encode what had lived only in engineers' heads, refreshed every few weeks by checking paths and re-running critic agents.
• Research that took about two days took about 30 minutes, with about 40% fewer tool calls (preliminary, six tasks).

Details on the rules file side: Rules and context.

The pattern: index, log, lint

• One index.md links every page; an append-only log.md records what changed; a periodic lint looks for contradictions between pages, stale claims and orphan pages (Karpathy, LLM wiki).
• Plain files in git: every major agent reads AGENTS.md (agents.md), and review works as it does for code (Write the Docs).
• People can browse the linked pages in any editor; Obsidian is one viewer that shows hubs and orphans as a graph.
• Agents propose page edits, an owner approves: wikis die when upkeep outgrows value, and agents do the bookkeeping.

Keep it short and specific

• Generic context files add over 20% to model cost with no general gain in task success; they help for non-standard practices (Gloaguen et al.).
• Files written by developers beat model-written ones by about 4 points of task success (same study).
• So pages stay short, one topic each, and an owner removes pages that do not change the agent's work.

Where it fits in the AI-SDLC

• Discovery: an inventory of what the team knows and where it lives.
• Implementation: the repository is set up with the pilot team in the guardrails step.
• Handover: its owner approves page edits; the monthly review reads the stale-page count and the questions agents had to ask a person.

The PRD next to it

docs/prd/DS-142-webhook-retry.md lists what must be true when the ticket is done, each line with the check that proves it.

• Line 1: WHEN no 2xx answer THE SYSTEM SHALL retry up to 8 times (check: npm test -- webhook).
• Line 2: the wait never exceeds 30 s.
• Four more lines follow, each with its check.

Each PRD line becomes a task on the board (Goal convergence).

PRD template

File docs/prd/<ticket>-<slug>.md, one page. Sections: Status, Owner, Decisions (the ADRs that bind it), Problem, Users, Scope (the paths the agent may edit), Acceptance checks, Out of scope, Open questions.

• Each acceptance check is one behaviour in one fixed sentence shape: WHEN a condition, THE SYSTEM SHALL a result (the EARS form, plain enough that a tester can check it).
• It ends with the Check that proves it: a command, a test or a query.
• A line with no Check is rewritten before work starts.
• The agent writes each open question with its recommended answer and the rejected option; the owner accepts or changes it, and an answer that sets a rule becomes an ADR.

An open question names an owner and the line it blocks. The agent stops on that line and asks; it does not guess.

ADR template

File docs/adr/NNNN-<slug>.md, next unused number, never reused. Sections: Status (proposed, accepted, rejected, deprecated, superseded by ADR-NNNN), Decided by, Context, Decision, Options considered, Consequences, Confirmation.

• Confirmation is required: it is the test or review question that shows the code still follows the decision.
• An accepted ADR changes only its status line.
• Reversing a decision adds a new ADR and marks the old one superseded.

How agents follow it

The repository rules file carries four lines:

1. Read the PRD and every accepted ADR before editing.
2. Work only on files in the PRD's Scope.
3. If a change contradicts an accepted ADR, stop and report the ADR number and the conflicting line.
4. An answer from a person that sets a rule becomes a new ADR in the same pull request.

• Each acceptance check becomes a task on the board (DS-142 gives six); the goal closes when all six pass together.
• A separate reviewer agent checks the diff against the PRD (Checks, Scope, Out of scope) and each ADR's Confirmation.
• A person still merges.

Where it fits in the AI-SDLC

• Choose the delivery process (Discovery and analysis) records the choice as the first ADR.
• Set up the delivery process (Implementation) puts both templates in docs/prd and docs/adr and adds the rules file lines and the reviewer check.
• Each roadmap task then starts as a PRD.

Client cases, anonymised (our measurement), and public reports

Test-automation SaaS company, ticket agent.

• The agent posted the same failure into one ticket three times and kept going.
• The written "stop after three failures" rule never fired.
• Fix: one failure counter per ticket, and dispatch stops when daily use passes a set limit.

Enterprise device maker.

• Agents could drift in quality and cost.
• Fix: three failures call a person.
• An agent never merges or approves its own work.
• Roll back when cost passes 150% of budget.
• Stop when complexity grows 40%.
• These limits are a design, not a measured result.

Cloud services provider.

• An agent fixing a bug rebuilt a live environment and caused a 13-hour outage of one service (press report; the company calls it user error).
• Fix: a person approves before any production change.

The permissions

• Merge: an agent never merges or approves its own pull request.
• Push: an agent commits only to its own branch; a person or a named service account opens the pull request under branch protection.
• Protected paths: CI workflows, lint, test, scanner and coverage config, infrastructure, secrets and payment code need the code owner's approval; gate config also needs the platform owner.
• Tickets: an agent attaches the passing check's output; a person closes the ticket.
• Priority: only a person raises a ticket's priority.
• Git: no force push or history rewrite on shared branches.
• Safety: back up or commit untracked work before an agent edits it; agents never print secrets in logs or diagnostics.
• Failures: after three failed attempts the agent stops and calls a person.
• Budget: the agent stops when its spend passes the set limit.
• Riskiest code: changes to login, cryptography and billing need reviews from two people.

Untrusted input. Ticket text, pull request descriptions, comments and dependency files can carry hidden instructions.

• The agent that reads them holds no secrets.
• It writes only to its own branch.
• Pull requests from forks are reviewed without secrets.

Sources: OWASP LLM01, GitHub Security Lab: preventing pwn requests.

The platform team owns the permission file; an engineer's local settings cannot widen it.

A rule is text the agent reads and can work around, so the hard limits sit in its environment: many cloud agents run at once, each task in its own sandbox (an isolated machine or worktree) that holds no production keys, and laptops running agents get none either and reach only approved hosts, and it never merges on its own.

Example: the orchestrator Cezar of Open Mercato (open-source project) runs each task in its own worktree; its README says "Nothing merges on its own" (README).

The problem

A strong model on every step overpays for easy work; a single provider that goes down or rate-limits stops the pipeline.

How the router works

• Model choice. The task-to-model list below; each task has a priority list of models.
• Fallbacks. On errors, rate limits, context-length limits or downtime the next model on the list answers (OpenRouter fallbacks).
• Provider rules. Order, price, throughput or latency, a price ceiling, and a no-data-retention requirement (OpenRouter provider selection).
• Cost caps. A spend limit per key and per team, with alerts before the cap is reached.

The task-to-model list

One list sets which model does each step; Model routing and Multi-agent follow the same list.

• Routine steps (summaries, formatting, ticket drafts): a small model.
• Writing the change: the main model.
• Critical review and cross-review: the strong model; cross-review uses another vendor's model.
• Any other step uses the strong model only when the list is changed to name it; nothing is upgraded on the fly.
• Logged: every answer records the model that gave it; a stand-in model is flagged, and strict mode refuses it on a check.

Your team edits the list; cost per change per model shows where to move work.

Sourced behaviour

By default the router sends most traffic to cheaper providers, and a provider with an outage in the last 30 seconds goes to the back of the line (OpenRouter).

Lesson. With your own provider key, keep the request pinned to that provider: a fallback to a paid route needs credit on the router account, and without it the call is refused.

Where it sits in the AI-SDLC

• Implementation: the router and the model list per task go into your agent setup.
• Measure and hand over: Cost per change per model shows where to move work; your team edits the list.

The problem

One vendor, one agent, one point of failure: a usage limit stops the work.

How the agents split the work

• Claude does the main work in the repository (Claude Code subagents).
• An approved second tool under your own account continues when Claude reaches its limit, tested and logged before you rely on it. Codex and Gemini are examples, not dependencies (Codex CLI, Gemini CLI); a vendor can withdraw a tool at short notice, so no step depends on one.
• Fallback chains. A usage limit on one tool falls to the next instead of stopping work.

The task-to-model list

One list sets which model does each step; Model routing and Multi-agent follow the same list.

• Routine steps (summaries, formatting, ticket drafts): a small model.
• Writing the change: the main model.
• Critical review and cross-review: the strong model; cross-review uses another vendor's model.
• Any other step uses the strong model only when the list is changed to name it; nothing is upgraded on the fly.
• Logged: every answer records the model that gave it; a stand-in model is flagged, and strict mode refuses it on a check.

Your team edits the list; cost per change per model shows where to move work.

Lesson: the silent fallback

• Problem: a usage limit on Codex silently fell through to Gemini, so one model answered both "independent" checks.
• Fix: every answer is logged with the model that gave it; a substitution is flagged, and strict mode refuses it.

Where it sits in the AI-SDLC

Implementation: fallback chains and the model log go into your agent setup. Measure and hand over: the log shows which model answered each check, and your team owns the chains.

How it works

• Why a command
  • The implementer runs it and sees the problem; without it, AI judges reject under 25% of invalid output
  • No command: marked judgement, cites a rule or ADR, a person decides, it cannot block
  arXiv 2510.11822
• Why three rounds
  • Five rounds of "improve this code" raised critical flaws 37.6%
  arXiv 2506.11022
• Rules
  • The tool refuses a round where implementer and reviewer are the same agent
  • Extra work the implementer spots goes to the reviewer as a numbered question; the reviewer rules on it

Where it fits in the AI-SDLC

Implementation. Each change in the pilot is built by one agent and checked by another before a person approves it, so the before/after in Measure and hand over counts verified work only.

Result

In a test on research findings: an automatic check accepted 22 findings once each carried its source reference; the reviewer read 6 by hand and found none supported by its source. The check alone would have shipped all 22.

• A separate reviewer asked why a function silently returned an empty result when a required input was missing.

When it does not help

A reviewer copied from the implementer's session shares its blind spots. With one actor only, the substitute is a review by a different model.

How it works

Two modes: on your laptop you type the command before you push; on pull requests a cloud agent starts it automatically and flags mismatches for a person.

One command sends a review, diagnosis or research task to another model family: Gemini, Codex or OpenRouter models.

The reviewer sees only the change and the project rules, not the author's reasoning.

A record names the model that actually answered and flags a substitution.

Naming and style run in the linter first (ESLint camelcase, Ruff pep8-naming); the second model gets what linters cannot see.

Findings from both models are combined; only a finding backed by a failing test, command or scanner line can block.

Where it fits in the AI-SDLC

Implementation. Every pilot change gets a second-model review before a person approves it; findings feed the Code Review area.

Result

Enterprise device maker. A second model caught a camelCase/snake_case mismatch the author model missed.

• A model reviewing its own output reads about 70% less before editing; we review with a second model instead.

• Combining several models raised false-alarm removal from 62.5% to 78.9% on a benchmark, but only from 33.85% to 38.46% on real code, so the real-code gain is small (arXiv 2506.16899).
• Combining the results of several review runs raised the F1 score (a mix of missed and false findings) by up to 43.67% (arXiv 2509.01494).

When it does not help

• A silent stand-in: the same model answering twice is not a cross-check.
• Unchecked answers once produced five confident findings about code that did not exist.
• The record of which model answered, and a check of the examined lines, catch this.

Checks before a person reads

• 1 Build, types, lint: fails the check when: any build or type error; a new lint error
• 2 Code rules: fails the check when: a new function, or one this change raised, over 15 in cognitive complexity (new code only, as in SonarQube); function over 50 lines or 5 parameters; over 3% new lines copied; new dead code or unused package; an import across a layer; a broken team design rule
• 3 Tests: fails the check when: changed lines under 80% tested; changed files' mutation score below last night's, or a mandatory planted bug open (Google)
• 4 Security: fails the check when: new high or critical finding; a secret; a vulnerable, missing or newly registered package
• 5 AI review: fails the check when: the finding quotes the line and a command or failing test proves the bug; otherwise it is "judgement" and cannot block
• 6 Low noise: fails the check when: over 400 changed lines; files outside the ticket

Client cases, anonymised (our measurement), and public reports

Five-person cloud-cost consultancy (several changes at once).

• 60 pull requests were open, two days of review for five people.
• Fix: a script labels changes that touch the public API, login, the design system, the database schema or agent skills; only those need a person.
• Shown for the queue effect; in our package a person still approves every product change.
• Lint rules, an 85% coverage floor and end-to-end tests came at the same time.
• Merged pull requests rose from 80 to 154 a week (self-reported).

European fashion e-commerce platform. Low-risk pull requests queued behind risky ones. A review bot handles the low-risk third (33%); their lead time is 20% to 40% shorter than that of all pull requests. In our package a person still approves every product change.

Test-automation SaaS company, review bot.

• 46 bot comments were 28 distinct findings, and half cited the wrong line.
• Fix: a gate drops a finding whose quoted code is not at the cited line.

Enterprise device maker.

• Four review layers (domain, tests, architecture, good practice) caught three runtime bugs in a 121-file pull request before merge.
• A design touching over 20 files gets a plan that splits it into ordered pull requests.
• Review comments are questions, one concern each, placed inline on the code line.

Review budget

Set before agents start writing: defect finding drops beyond about 400 lines and after about 60 minutes (SmartBear). Without a budget, agent output piles up unread and approvals become a formality.

• Pull request size
  • At most 400 changed lines
  • Bigger: split, never waved through
• Review session
  • 60 minutes
• Open reviews
  • Capped per reviewer
• Who opens and merges
  • The agent commits to its own branch
  • A person or a named service account opens the pull request under branch protection
  • A person approves the merge

Why each check, measured

• Tests at 100% coverage caught 4% of planted bugs: diff coverage needs a mutation score beside it (arXiv 2506.02954).
• AI co-written pull requests drew 1.7x the review findings: 10.83 against 6.45 per pull request (CodeRabbit, vendor).
• Model judges accept over 96% of valid output and reject under 25% of invalid output: an AI finding needs a command that shows it (arXiv 2510.11822).
• Merging several AI review runs raised F1 (found bugs against false alarms) by up to 43.67% (arXiv 2509.01494).
• 5.2% (commercial models) to 21.7% (open models) of package names were invented (arXiv 2406.10279).
• Critical flaws rose 37.6% after five rounds of "improve this code" (arXiv 2506.11022).
• AI triage removed 34% to 38% of scanner false alarms on real code (arXiv 2506.16899).

AI review rules

• Starts after checks 1 to 4 pass, reads their output, never repeats them.
• Two or more runs from different model families, because a model rates its own kind of output higher (arXiv 2410.21819); the union of findings is kept.
• CI runs the AI review on the pull request's latest commit, signs the output with the commit id using GitHub artifact attestations, and the summary check fails without it.
• Each finding carries a command or failing test that shows it; without one it is marked "judgement" and cannot block alone.
• An AI check blocks only after 90% precision and 60% recall on 30 past changes, labelled by two people (agreement measured) and not used to tune it (held out); until then it warns. Five bots scored 64% to 96% precision (Signal65, vendor-commissioned).
• Risk label high: the change touches login, payments, data migrations, CI settings, dependency files or a code-owner path, or carries a scanner finding or a new package; the code owner approves.

Methods and tools

• Types: TypeScript strict, mypy, pyright.
• Linters: ESLint (max-lines-per-function), Ruff (too-many-arguments), golangci-lint.
• Cognitive complexity: SonarQube rule S3776, default limit 15 (method, why 15).; Biome noExcessiveCognitiveComplexity for JS/TS, fast in biome ci.
• Quality gate on new code: SonarQube quality gates, duplicated lines on new code at most 3%.
• Duplication: jscpd, PMD CPD.
• Dead code, unused packages: knip (JavaScript, TypeScript), vulture (Python).
• Layers and module boundaries: ArchUnit (Java), dependency-cruiser (JavaScript), import-linter (Python).
• Team design rules as code: Semgrep, ast-grep.
• Tests on changed lines: diff-cover; mutation testing with Stryker, PIT, mutmut.
• Security: CodeQL, Semgrep, gitleaks, GitHub dependency review, OSV-Scanner, package age and origin on deps.dev.
• Size and noise: Danger fails a pull request over the size limit or touching files outside the ticket.

Thresholds are starting values, set with you; the gate table is in Quality gates.

Try the change first

Where the team has preview environments, a change to a screen or an API opens one, and the reviewer uses it before reading the code.

Numbers to expect

About half of agent patches that pass the tests would still not be merged by a maintainer; the maintainer merge rate is 24.2 points below the automated score (METR).

In a year-long deployment of AI code review, cycle time fell 30.8% and 38.70% of review comments were acted on (arXiv 2601.01129). Expect less waiting, not every comment fixed.

Industry

AI-written pull requests wait about 16 hours for a first review, against about 200 minutes for others, across 8.1 million pull requests (LinearB 2026, vendor).

Open Mercato (open-source project) says "100% of code is generated by coding agents" (HIRING.md); people still own the architecture, the specs and every merge, and an audit of it found 16 high-severity flaws.

How the mutation score is run

• Changed, covered lines only, one planted bug per line, 10-minute limit (Google, CI); over it, "not measured"
• No fixed floor: none is published; 60% is Stryker's red colour (Stryker)

Client cases, anonymised (our measurement), and public reports

Search and cloud company, bug-fixing agent.

• For machine-reported bugs, 73% of agent patches passed the tests, but only 43% were correct when checked by hand (25.6% and 17.9% for human-reported bugs).
• Fix: validate a patch before showing it and hold back the ones unlikely to be accepted.
• Validation and holding patches back raised usable results by up to 29 points in the published paper (39 in the preprint).

Internal AI-tooling team.

• One check run returned 15 tidy verdicts, 11 of them wrong, while every check passed.
• Fix: each test must fail on a planted bug before its result is trusted.

Ride-hailing and delivery platform. Generated tests worked for 80% of Python code but 20% of Java. Fix: measure per language, not one headline rate.

Fixing a bug: test first

Incident tests go through the same before/after run.

Tests cannot be weakened

• The gate fails on a new skip, only or xfail marker (no-focused-tests, pytest --strict-markers).
• It fails on a snapshot rewritten in CI (jest --ci) and on fewer tests than the merge base.
• Deleting a test file needs the code owner.
• Parsers, validation and login code get property tests (Hypothesis, fast-check, jqwik) or a bounded fuzz run; any counter-example fails.

Why coverage is not enough

Line coverage of 90% can sit beside tests that catch nothing: a suite can reach 100% coverage and catch only 4% of planted bugs (arXiv 2506.02954). Test strength counts the planted bugs a test catches. The scores in the demo above are demo values, not measurements.

• Where bugs are planted
  • Changed, covered lines only, one per line: Google cut the median from 820 to 7 planted bugs per change
  • A 10-minute run, the usual CI build limit; over it, "not measured", never "passed"
  Petrović et al.Fowler
• Surviving planted bugs
  • Each becomes a review comment, at most 7 per file; Google engineers rated 75% useful
  ICSE-SEIP 2018
• Blocks the merge
  • A planted bug the reviewer made mandatory
  • Changed files score below last night's run on main (the coverage rule)
  Codecov
• No fixed floor
  • None is published; Stryker's 60% only colours the report unless break is set
  • At Facebook over half the planted bugs survived
  • Set break at your own nightly score
  Stryker thresholdsBeller et al.
• Coverage gate
  • Under 80% of changed lines and branches run by tests
  • Total coverage drops
  diff-cover over JaCoCo, Istanbul or coverage.py
• After a change or an incident
  • Agent or harness change: one real end-to-end case runs on the real path
  • Production incident: it leaves a test, so the failure cannot return unnoticed
  • Each correction: the agent updates one record in an indexed lessons catalog, checked in CI
  Open Mercato (open-source project): 152 lessons

Social network company: 73% of tests generated for privacy testing were accepted by its engineers in a trial (published 2025).

Why it matters

Known-exploited flaws took a median 43 days from detection to full fix; only 26% were fully fixed (Verizon DBIR 2026). Fewer than 1 in 10 dependency flaws can be reached from the application's code (Endor Labs, vendor data), so a reachability check cuts the alerts to the few services that need a fix. We measure your share in Discovery and analysis.

• Reachable or unproven, high or critical
  • Blocks the merge
  govulncheck (Go)OSV-Scanner call analysis (Go, Rust)
• Proven unreachable
  • A ticket due in 30 days
• Medium
  • An owner and a due date
• No fixed version yet
  • The agent lists the exposed services
  • It proposes a temporary block until the fix ships: switch off the feature, change a setting or add a firewall rule
• Why teams welcome it
  • Removes an unplanned day of searching and upgrading
  • A person still approves
  • Reuses your gates, merge line and approvals
  • A team that sees it work trusts agents with the next step
• Limits
  • Your release process still deploys
  • The upgrade is only as safe as your tests (Testing)
  • Every fix has a revert (CI on main)

Client cases, anonymised (our measurement), and public reports

Test-automation SaaS company.

• 737 security tickets were open across five projects, the oldest over two years.
• The written rule (no Medium or higher merges) differed from the gate that ran (Critical only).
• Fix: sweep and close done-but-open tickets, triage new reports first, and make the written gate match the one that runs.

Large internet company. 76% of the alarms in a 433-alarm study set were false. A hybrid of model and static analysis removed 94-98% of the false alarms, with recall of 0.75-0.88: some real bugs were filtered out too.

Search and cloud company. An agent that drafts fixes opened 72 security fixes in six months; a person reviewed every one before it landed.

The example

• Spring4Shell: data binding can reach the class loader and let an attacker run code.
• The order controller binds request fields, so the path is reachable.
• The scanner decides; the model explains the finding in the pull request.

AI-generated code failed security tests in 45% of samples across 100+ models (Veracode 2025, vendor).

The EU Cyber Resilience Act sets reporting and vulnerability-handling duties for products with software (European Commission).

Code scan

• CodeQL security-extended or Semgrep runs on every pull request against the merge base.
• A new high or critical finding on changed lines blocks the merge.
• AI triage may lower a finding's rank, never dismiss it: the best model removed about 62.5% of false alarms, not all (arXiv 2506.16899).

Supply chain and release

• Lockfile
  • Required; CI installs with npm ci or uv sync --locked
• New package
  • Must exist in its registry and be over 30 days old
  • A new install script is flagged
  deps.devSocket
• GitHub Actions
  • Not pinned to a commit: the gate fails, as a tag can be moved (GitHub)
  zizmor
• Licences
  • Off the allow list or unknown: fails, on every pull request
  • The full tree is scanned before each release
  dependency review actionORTScanCode
• Release
  • Ships a CycloneDX or SPDX bill of materials, scanned
  • A high or critical flaw with a fix blocks the release
  • Carries a build provenance attestation (SLSA Build L2)
  syftgrypeTrivyGitHub artifact attestations
• Upgrades
  • Patch and minor: one approver after a green gate
  • Major: the code owner
  • Each ships with its revert

Secret scan

• gitleaks runs as a pre-commit hook in every agent sandbox and in CI; GitHub push protection is on; full history is scanned at onboarding.
• A key in the diff blocks the merge.
• The agent moves the key to an environment variable and rewrites its own unmerged branch, so the key is in no commit.
• The key's owner rotates it within 24 hours.

Gate table

• Build and types: fails when: any error; starts as: block from day one
• Linter: fails when: one new error; starts as: block from day one
• Cognitive complexity: fails when: new function, or one this change raised, over 15 (new code only, as in SonarQube); starts as: warn first
• Duplication: fails when: over 3% of new production lines copied; test files reported apart; starts as: warn first
• Architecture rules: fails when: one forbidden import; starts as: warn first
• Dead code: fails when: the change adds any; starts as: warn first
• Diff coverage: fails when: under 80% covered; starts as: warn first
• Mutation score on changed lines: fails when: changed files score below last night's main run (as for coverage), or a mandatory survivor is open; over 10 minutes: measured nightly, a warning on the pull request, not a skip: "not measured", never "passed"; starts as: warn first
• Code security: fails when: new critical or high finding; starts as: block from day one
• Secrets: fails when: any secret; starts as: block from day one
• Vulnerable dependency: fails when: high or critical flaw; starts as: block from day one
• New package: fails when: name missing (invented), under 30 days old, or unreviewed; starts as: block from day one
• Licence: fails when: not on the allow list; starts as: block from day one
• Findings per AI round: fails when: any count rises, or a fourth round starts (arXiv 2506.11022); starts as: block from day one
• Tests not weakened: fails when: fewer tests, a new skip, or a snapshot rewritten without the code owner; starts as: block from day one
• Gate config protected: fails when: edited by an agent, a gate rule switched off, or edited by a person without the platform owner; starts as: block from day one
• Pull request size: fails when: over 400 changed lines; starts as: warn first

Thresholds are starting values, set with you. A warn-first gate reports on the pull request and blocks once false alarms are rare.

Client cases, anonymised (our measurement), and public reports

Enterprise device maker, scanner evaluation software.

• A misspelt prediction key scored every barcode dataset at zero.
• Fix: one gate command (lint, tests, coverage, security, branch name, title) runs before a pull request opens.
• A person approves design, code and merge.
• On that ticket the gate passed 997 tests at 93% coverage before a person saw the pull request.

Same company, agent pull requests. The agent's pull request runs the same gate as a person's, on every push, so nothing passes only on a laptop.

Search and cloud company, code migration. A migration of 5,359 files over three months: machine checks decided what passed, and 87% of the AI-written code landed unchanged.

Prove the checker

A gate is only as good as the check behind it. Four habits prove an AI check before it may block a merge:

• Calibration. The check grades a sample a person also grades. An AI check blocks only after 90% precision and 60% recall on 30 past changes, labelled by two people (agreement measured) and not used to tune it (held out); until then it warns. On code, model judges accept over 96% of valid output and reject under 25% of invalid output (arXiv 2510.11822).
• Replay tickets. Past defects from the domain expert rerun before a rule, prompt or model changes; the change waits until they pass (Anthropic on agent evals).
• Pre-flight. Tool servers, model endpoints and judges are checked before a run, and the AI reviewer must find one planted bug, so a reviewer that stopped catching bugs grades nothing (planted bugs, Google); an unready tool stops the run and is not counted as an agent failure.
• Model version log. Every answer records the model that gave it, so a silent model change shows (OpenTelemetry GenAI conventions).

Security gates, set-up

• Code scan: CodeQL with the security-extended queries, plus Semgrep rules for your own security rules; new critical or high fails.
• Secrets: gitleaks on every pull request and as a laptop pre-commit hook; any hit fails and the secret is rotated.
• Dependencies: dependency review action with fail-on-severity high and a licence allow list; OSV-Scanner for lock files outside GitHub.
• Invented packages: each new name is looked up in its registry and on deps.dev; missing or under 30 days old fails. Models invent 5.2% to 21.7% of package names (arXiv 2406.10279).
• Findings per round: scanner counts are kept per agent iteration; a rise fails. Critical flaws rose 37.6% after five "improve" rounds (arXiv 2506.11022).

Methods and tools

• Gate engine: SonarQube quality gates on new code; GitHub rulesets make each check required.
• Cognitive complexity: SonarQube rule S3776, limit 15 (method, why 15).; Biome noExcessiveCognitiveComplexity for JS/TS, fast in biome ci.
• Duplication: jscpd, PMD CPD.
• Architecture: ArchUnit, dependency-cruiser, import-linter; team design rules in Semgrep or ast-grep.
• Dead code: knip, vulture.
• Tests: diff-cover; Stryker, PIT, mutmut. Tests at 100% coverage caught 4% of planted bugs (arXiv 2506.02954).
• Security: CodeQL, gitleaks, GitHub dependency review, OSV-Scanner.
• Size: Danger.

Client cases, anonymised (our measurement), and public reports

Test-automation SaaS company.

• CI broke the same few ways after each platform migration, and each team rediscovered the fix alone.
• Fix: a red pipeline is matched to a break already fixed elsewhere, and the same fix arrives as a pull request.

Same company, reverts. One team reverted three dependency bumps from production in three months. Fix: every break lands as a pull request with its own revert.

Ride-hailing and delivery platform. An agent reads the failed job and opens the fix; a person reviews it.

The rule

The repair agent starts read-only: it explains each failed build on the pull request and flags flaky tests (Google on flaky tests). It starts fixing once its explanations prove right.

Each break is fixed by a new pull request, never by a direct push. The 90% main-branch success threshold is checked on your own pipeline.

Industry: 70.8% main-branch success in 2026 and 72 minutes to get back to green (CircleCI 2026, vendor).

Client case, anonymised (our measurement)

Test-automation SaaS company.

• Of 5,297 resolved tickets, 38 (0.7%) carried a release version, so nobody could show what a customer received, agent-built changes included.
• Fix: the deploy pipeline writes the version onto the ticket.

Same company, agent register. Two agents running in production had no register entry. Fix: each agent gets a name, a budget and an owner.

Enterprise device maker. Every agent pull request carries its ticket in the branch and title, an ai-assisted label and a named approver, checked before the pull request opens.

The audit trace

• Kept per agent actionTicket idAgent and model nameTimeCommand or file changedResultApproverPull requestRelease version
• Agreed with your audit team in Phase 1
  • How long records are kept
  • How tampering is detected
  • Export format, to fit the tools you already audit with
• Set up in every case
  • The log sits where agents cannot write or delete
• What the trace gives you
  • Compliance: who approved each change, on record
  • Debugging: replay the agent steps, files changed and approvals to find the cause

Client cases, anonymised (our measurement), and public reports

Ride-hailing and delivery platform.

• Weekly users grew 7x and agent requests 9.4x.
• Fix: automatic compaction, medium reasoning effort by default, a one-hour prompt cache.
• Live session cost in the terminal; alerts at 50/80/100% of expected spend.
• With the model held fixed, cost per session fell 52% from its peak.

Same platform, tool calls. Each tool call cost one model round trip. Running batched scripts instead cut tokens by 55% to nearly 100% per query in a five-query test (the article says more than 50%).

Test-automation SaaS company.

• One unwatched agent session cost about twice the whole matching backlog in a day.
• A shared monthly budget drained in days with nobody told.
• Fix: a hard cap per agent account, with warnings before it.

The levers

• Prompt caching and short sessions: see Token savings.
• Batch processing: about half price for work that can wait.
• Model routing: a smaller model for easy steps, checked on your coding tasks first; public chat, knowledge and maths tests showed 35% to 85% lower cost at 95% of the large model's quality (RouteLLM).
• Open-weight models: no per-request fee; code stays in your network, and you pay for the hardware.
• Context control: less for the model to read.
• Team budgets: caps heavy users, with alerts before the cap.

Detail: Token savings (session and cache rules) and Model routing (model choice and fallbacks).

Why the bill grows

• Each call re-sends the whole conversation, so a long session re-reads every earlier turn.
• A pause longer than the cache lifetime throws the cached part away; it is paid again at full price.

The four rules

• Stable prompt first. Tools and rules first, the changing part last, tool results kept out of cached blocks (arXiv 2601.06007).
• One task, one session. Clear the context between tasks instead of carrying it (Claude Code costs).
• Handover note. The agent writes what is done and what is next; a fresh session reads the note, not the transcript.
• Cache lifetime fits pauses. A ride-hailing platform moved from a 5-minute to a 1-hour cache because idle gaps kept discarding it. Short sub-agent tasks kept 5 minutes (engineering blog).

Model choice and batch: Model routing and AI cost levers.

Sourced results

Published study. Caching cut cost 41% to 80% over 500+ agent sessions on four models (arXiv 2601.06007).

Security-tooling company. Static prompt on a 1-hour cache, conversation on a sliding window: cache hit rate rose from 7% to 84%, savings 59%, then 66%, then 70%.

Caution. Caching everything, tool results included, sometimes slowed execution.

Where it sits in the AI-SDLC

Implementation: session, handover and cache rules go into your agent setup. Measure and hand over: Cost per change shows the effect, and your team owns the rules file.

How it works

A script finds the padding; agents rewrite one section at a time.

Rules are split across specialist checks, because one agent holding 35 rules applies about four.

A check compares numbers, paths, identifiers and rules with the original and keeps the original until you confirm.

Where it fits in the AI-SDLC

Measure and hand over. The Discovery and analysis report, the roadmap and the before/after report reach your leads shortened to the target length.

Result

Runs land at 1.11 to 1.27 times the target length across 16 samples and four model backends; a second pass removes only 0% to 11% more (mean 6%, nine samples).

When it does not help

A document with under 5% padding: three runs on one 8,545-word log removed 3.3%, 0.6% and 0.3%.

Tool

kill-verbosity on GitHub

Discovery and analysis (Phase 1)

• Covers: discovery workshop, analysis of your tickets, git and CI (read-only), maturity workshop on 16 areas, choice of the delivery process.
• You get: report with the weakest area agreed, a roadmap with measurable milestones, the baseline of five numbers (lead time, started tickets that reached release, review wait, change failure rate and cost per change).
• Then you decide: stop with the report, run the roadmap yourselves (remote support optional), or continue with us.
• From us: Marek Koniew, or a collaborator working with him, leads every workshop and builds with the pilot team; every report is checked by a second model family, then by Marek Koniew.

AI-SDLC Package (Phases 1-3)

• Covers: Phase 1; Implementation in 2-3 areas (for example code review or testing) with one pilot team, guardrails first, weekly check-in (4-8 weeks); Measure and hand over.
• You get: report and roadmap, tools and practices fitted to your stack, changes running and tested in the agreed areas, the five numbers before and after, owners for each change.
• Implementation runs with our consultant inside your team, not as a report handed over.

Due diligence (Light or Full)

• Light (~1-2 weeks): architecture and code, for smaller or early-stage companies; a first view and whether Full is needed.
• Full (~2-4 weeks): a decision report with red flags and conditions; architecture, code quality and technical debt, team and its capacity to grow the product, scalability, risks (dependencies, vendor lock-in, IP, open-source licences).
• You get: a decision report with red flags and a recommendation (proceed, proceed with deal conditions, or stop); optional Q&A debrief with the deal team.
• Duration depends on access to the company's repositories and team.

How to read this list

Concepts are grouped by what they do: find where to start, run the work with agents, keep every change safe.

Sources: research and market data, standards, tools and methods, and cases by industry.

Measurement

The METR 2026 re-run was inconclusive.

Adoption: share of work agents already do

• Pragmatic Engineer: How Uber uses AI for development 11% of Uber's pull requests are opened by agents (March 2026).
• Google, Alphabet Q3 2024 earnings remarks More than a quarter of new code is generated by AI, then reviewed by engineers.
• Anthropic: recursive self-improvement More than 80% of merged code was authored by Claude (May 2026).

Release: code written is not code shipped

• DORA 2025: State of AI-assisted Software Development AI amplifies the strengths and weaknesses of the delivery system.
• Sadowski et al.: Modern Code Review at Google A case study of why and how review is done, over 9 million reviewed changes (written before AI coding).

Security: every agent change needs a check before merge

• Spracklen et al.: package hallucinations On average at least 5.2% (commercial) to 21.7% (open-source) of suggested package names were invented.
• Liu et al.: LiCoEval 0.88-2.01% of outputs closely match open-source code.

Measurement: only measured numbers count

• DORA software delivery metrics Lead time, deployment frequency, recovery time, change fail rate, rework rate.
• Forsgren et al.: The SPACE of Developer Productivity Productivity is not one metric.

Cost: predictable as use grows

• Chen, Zaharia, Zou: FrugalGPT A model cascade matches the best single model at much lower cost.
• OWASP LLM10:2025 Unbounded Consumption Quotas and anomaly detection against runaway use.

How the groups are built

Each line: the source, what it shows for the AI-SDLC, and where we act on it. Vendor reports are marked; preprints are research that is not yet peer-reviewed.

The market group lists analyst and industry sources only.

Why standards

A check that cites a standard is easier to approve: security, audit and works-council reviews read a shared vocabulary instead of our own terms.

Some ISO pages block automated readers; their content is confirmed from the standard's public summary.

Tool class per step, and the risk it reduces

• Ticket: specifications, against vague tickets.
• Plan: context repository, against guesswork.
• Build: sandboxed agents with a switch-off, against leaked keys and runaway agents.
• Review: AI and human review, against missed defects.
• Merge: quality gates, against flawed merges.
• Every step: prove the checker, against an AI check that is confidently wrong or a silent model change.
• Release: release record, against untraced changes.
• Run: production monitoring, against late incidents.
• Measure: the five numbers and cost per change, against gains claimed but not shown.

How we choose

We start from what your team already runs and add a tool only where a step has no check. Vendor tools are listed for what they do, not as a recommendation. No step depends on one vendor: the fallback is an approved second tool under your own account.

About these cases

Each case links its public source; numbers are self-reported. Our own client cases are described by industry; details on request.