What is a software factory?#
A software factory converts human intent into safe, measurable outcomes using people, agents, orchestration, context, environments, reusable components, validation, policy, governance and learning. It is deliberately broader than an autonomous coding environment, and the difference shows as soon as an organisation moves past individual assistants. An agent may implement a feature correctly and still pick the wrong service boundary, miss a tenant-isolation requirement, or produce a change nobody can operate. Each of those is a failure of the system around generation.
| Dimension | AI-assisted team | AI software factory |
|---|---|---|
| Execution | Engineers drive most steps, with tools assisting | Agents execute bounded workflows; people own intent and consequential decisions |
| Intent | Requirements interpreted per task, mostly in conversation | Intent becomes a specification with acceptance conditions validation can compare against |
| Context | People assemble it from code, documents and memory | Curated, versioned and served as infrastructure |
| Validation | Review plus CI, with humans judging most changes | Layered assurance at machine speed, human gates placed by risk |
| Comprehension | Understanding is a by-product of writing the code | Understanding has to be produced deliberately, because nobody wrote the code |
Differences in emphasis. Mature teams already do several of these and can extend into a factory incrementally.
DevOps supplied the practices, CI/CD the automated pipeline, platform engineering the reusable capabilities. A software factory assembles all three and adds agent execution. None of the older layers becomes optional; each becomes load-bearing in a way it was not before.
The term itself is forty years old, redefined three times before this one: industrialised process and reuse (1991), assembling applications from models (2004), and a hardened DevSecOps delivery organisation in US defence from 2017, where Platform One is the known example and the usage is still current. Each generation promised to move humans from writing software to supervising it, and each hit the same problem: supervision requires understanding, and understanding used to be a free by-product of doing the work.
Where does the bottleneck move in a software factory?#
The bottleneck moves, and in a predictable order — the pattern the operating-model series calls constraint migration. Throughput is set by the constraint, so improving any other station produces inventory instead of delivery. It is worth being precise about which loop is being improved: the build loop runs from a specification to a merged, tested change, and the delivery loop runs from an idea to a measured outcome. Almost all the waiting is in the second. That is Goldratt's argument applied to a delivery pipeline, and it is the organising idea of this guide.
Before AI, build was the constraint. It was slow and expensive, which is why everything around it was allowed to be slow: three months of requirements gathering did not matter when development also took three months. Solve build and that tolerance disappears, one station at a time.
The transitions are more useful than the states, because each is a symptom a leader can recognise without an assessment. Use spreads, and everyone works differently. Review is slow, because pull requests arrive faster than a fixed number of reviewers can read them. CI is not proof, because a green pipeline says nothing about whether the change does what was wanted. Context is missing, because an independent reviewer is only as good as the rules it holds. Parallel work arrives, and integration sets the pace. Runs repeat their mistakes, because nothing is fed back. And finally the machinery needs an owner, or it decays.
In the value-stream assessments we run, the calendar tells the same story. A change with two to three days of engineering work inside it routinely takes five to six months from idea to production, most of that waiting between stages. Compressing three days of engineering into an afternoon attacks about two days out of a hundred and sixty.
Two consequences follow. The right next investment is rarely the most advanced capability available: an organisation whose reviewers are drowning gains nothing from a knowledge graph. And once one station reaches machine speed the next has to follow, because a human in the performance path caps everything behind them.
What do you need in place before building a software factory?#
Cloud Native organisations get further, faster. Once the constraint has moved past generation, the strongest predictor of how much value an organisation gets from agents is whether it can already carry a change from commit to production without a human in the performance path. The choice of model matters much less.
Organisations that are genuinely Cloud Native have that. Their tests carry code to production without a manual testing stage, they deploy on demand, and they roll back automatically when production says something is wrong. Putting agents on top of that is comparatively straightforward. An organisation without it has to build it first, because a system that stops to wait for a person at every stage cannot absorb the speed agents are capable of. Plenty of engineering organisations still have one tester manually checking twenty things before a release; that will not survive contact with agents.
The uncomfortable part is that the prerequisite list is the continuous-delivery research of the last fifteen years, restated. Three of the seven capabilities in DORA's 2025 AI Capabilities Model are plain continuous delivery: strong version control with frequent rollback, small batches, and quality internal platforms. The load-bearing finding underneath is about approval, where DORA reports that "no evidence was found to support the hypothesis that a more formal, external review process was associated with lower change fail rates". The cost of an approval gate has been measured; no offsetting benefit has been. That asymmetry worsens as volume rises, which argues for replacing human gates with automated gates plus fast recovery. It does not argue for removing review.
Two objections are worth keeping in view. The suite may not be the gate you think it is: Google reports that "about 84% of the transitions we observe from pass to fail involve a flaky test". And you can use agents to build the foundation, as Meta and Google both have, though in each case inside a validation harness rather than instead of one.
Why do software factories need specs first?#
Specify before you accelerate. Where engineering work is two or three days inside a five-month lead time, the first investment belongs upstream. Mining 921 requirement-emergence events across 402 real coding-agent sessions found that "a requirement's arrival is followed by roughly twice as much invalidation as matched non-requirement edits", with no decline over a session (Jiang et al., 2 September 2026). Ambiguity costs too: the same ambiguous requirement reliably produces functionally different programs on different runs.
The recommendation we most often make after an assessment is to put AI into product discovery before implementation. Product people can build working prototypes grounded in what the system already does, show a customer something real in days, and produce specifications describing behaviour someone has reacted to. The classic caveat holds: Boehm's 1984 experiment found prototyping teams produced 40% less code for 45% less effort, and lower robustness, because a prototype expresses no non-functional requirements.
A specification is worth writing to the extent that validation can compare against it. "Create a tenant onboarding API" leaves every consequential choice open; a usable one states who may call it, what makes it idempotent, how tenant boundaries are enforced, what happens when provisioning partially fails, and which service owns the data. Spec-driven development as a named practice is not settled, though: Thoughtworks places it in Assess, and the first academic treatment says the evidence base does not exist.
How do you give agents the right context?#
The right knowledge, not all of it. Validation needs knowledge, so the constraint moves upstream from the checks to the context that makes them meaningful. The goal is the smallest set that works. Anthropic's framing is the clearest: models have "an 'attention budget'" and every token spends it, so the target is "the smallest possible set of high-signal tokens" (Anthropic, 29 September 2025). Chroma's evaluation of 18 models found that "performance grows increasingly unreliable as input length grows", on tasks that are trivial at short length.
Documentation written for humans fails as machine context in two ways: it goes stale, and it multiplies. If seven descriptions of the same architecture exist across a wiki, a README and two decision records, a model cannot tell which is true. Codifying context into files the agents read solves the addressing problem and inherits the maintenance one: across 100 repositories using agent instruction files, only nine were free of any identified smell.
The format has settled further than the behaviour. AGENTS.md, MCP and goose were
donated to a Linux Foundation body in
December 2025.
Governance consolidated; behaviour did not. Claude Code still reads CLAUDE.md and documents importing
or symlinking AGENTS.md as the interop route, Cursor reads its own rules directory, Copilot its own
instructions file. One canonical file plus thin per-tool shims is what survives that.
Knowledge graphs are the deeper direction, and the evidence is more mixed than the pitch. A graph is genuinely good at decisions and their supersession, at provenance chains, and at invalidating facts that are no longer true. It is not reliably better at finding code. Comparing the families across 427 samples and 25 repositories, "no retrieval family dominates across workflow signals or context budgets": embeddings lead on recall, structure-aware retrieval on how much useful context fits a fixed token budget and on tracing a failure back to the file that should change (Qin and Xie, 27 July 2026). The same work found agent trajectories retrieving none of the required files on 27% to 35% of samples, which is the number to fix before choosing an architecture.
Shared context is also a shared failure mode. Engineers are heterogeneous, and that variation acts as a set of circuit breakers; a single shared factory has none, so what enters shared state needs a review stage of its own.
Why isn't passing CI enough to trust agent-written code?#
Green CI is not proof. Validation asks two questions, and the second is the hard one: does it work, and is it what was wanted? The standards separate them by a single clause, verification confirming that specified requirements have been fulfilled and validation that the requirements for the intended use have been (ISO/IEC/IEEE 12207-2008). Checks verify form, validation checks intent, and a green pipeline is evidence only about the first.
That gap is measurable. Analysis of patches passing SWE-bench's tests found 29.6% of them behaving differently from the accepted fix while passing every test. Agents also game the checks: OpenAI's catalogue of coding reward hacks includes "modifying functions upstream of the testing framework and unit test files such that tests trivially pass". Deliberate tampering is rare, though; the common failure is a route that passes the visible checks without generalising.
Two properties matter more than the exact layers. Repairs are bounded, because an agent with unlimited attempts eventually finds a route that satisfies the check without satisfying the requirement, and every repair re-enters the wall from the bottom. And the top layer has entry conditions of its own, so a human is called in at a defined point instead of catching whatever the other five missed.
Deterministic gates, and their limits
Some gates must be deterministic and blocking, because compliance is binary and a probabilistic judgement is not an acceptable answer. You cannot rely on a language model for that, and you cannot rely on a person manually testing twenty things either, because that person is now the constraint. The qualification is proportion: most tests will be machine-written, and the deterministic set is the smaller layer that prevents disaster. It is narrower than it looks, too: deterministic SAST blocks injection classes and is structurally blind to authorisation logic, which OWASP states plainly in listing access control among what these tools cannot automate. Their other cost is noise: in a Tencent deployment "the proportion of false positives was at 76%", above 90% once inaccessible cases are counted, while an LLM triage layer over the same warnings removed 94% to 98% of them (Du et al., 26 January 2026). The model reading the scanner output may be worth more than the scanner. Every gate should state its own scope.
The techniques that beat tests at their own job all work by giving you an oracle you did not write by hand, and the reference stack below lists them. The one worth revising an old assumption about is bounded model checking, now cheap enough to gate on rather than admire: Kani runs "over 16,000 harnesses verified per code change" in the Rust standard library campaign (Delmas et al., 1 July 2026).
Be careful what you point the agent at. Coverage is the obvious wrong target: control for suite size and its correlation with effectiveness largely disappears, while assertion count holds. Mutation score is the less obvious one. A July 2026 replication found both informative for comparing models on code that can be assumed correct, and unreliable where "the code-under-test may already be buggy and the goal is to expose the bug" (Zhao, Zhou and Cohen, 24 July 2026). That is the factory case, so assertion density survives as a target and mutation score is better read as a diagnostic.
A second model call is not a second opinion
Never let the agent that wrote the code decide whether it is acceptable: same context, same assumptions, same blind spots. Adding another model buys less than it looks, since nine frontier judges from seven model families proved worth roughly 2.18 effective independent votes. Independence is a property of architecture, not headcount: a real check gets fresh context assembled from the specification, an adversarial brief, and restricted tools with no write access to the acceptance evidence. Keeping end-to-end scenarios outside the codebase the agent can see, the way a machine-learning team keeps a holdout set, is the practice that follows.
How much autonomy should an agent get?#
Autonomy runs as a set of modes at once, one per class of change, because the consequence of a mistake differs enormously between a documentation update and a database migration. The slider from supervised to unsupervised is the wrong mental model. The written form of those modes is a delegation boundary: what the agent may do without asking, when it must stop, who is asked, what evidence comes with the ask, and what happens if nobody answers. Sam Newman makes the case for keeping limits in place from the operations side in his episode on why AI cannot fix your production incidents yet.
| Class of change | Mode | Why |
|---|---|---|
| Documentation | Auto-merge after validation | Cheap to reverse, and the checks are meaningful |
| Application code | A human approves the pull request | One reviewer, with the tier set by risk |
| Migrations | Two-key authorisation | Irreversible. NIST SP 800-53 CM-5(4) and PCI DSS already require this shape |
| Regulated | Never unattended | A legitimate end state, and not a failure to mature |
None of this is new and should not be presented as new: levels of automation applied per stage have been the standard human-factors model since Parasuraman, Sheridan and Wickens in 2000. Vendors ship the same shape now. In Anthropic's permission model no allow rule and no hook can approve a deletion targeting a critical path, because "This circuit breaker guards against model error", and the protected paths include the hook configuration files themselves (Claude Code). Shielding the gate infrastructure from the agent is the decision we would protect first.
June 2026 showed why. The Miasma worm reached 73 Microsoft-affiliated repositories and executed through
agent configuration rather than a dependency, planting session hooks in .claude/settings.json and
.gemini/settings.json, a rule file under .cursor/rules/, a VS Code task and an npm script, so one payload
ran from five developer tools
(SafeDep, June 2026). Agent configuration
is executable supply-chain surface and needs the review and protected-path treatment CI configuration gets.
Sandboxes now say so: Claude Code denies writes to the files
it loads its own configuration from, because a command that could edit them "could grant itself permissions,
or add a hook or MCP server that Claude Code runs outside the sandbox". Authorship is the gap still open, and
SLSA v1.2's Source Track, added in November 2025, is where a durable answer to
"did anyone review this" belongs.
Two rules make the modes work. Put the controls before the autonomous part and make them blocking, because once a system is running there is limited opportunity to interfere, in the same way that once you stop driving you are no longer in the loop that steers. And a human gate earns its place only when judgement is genuinely required.
What does a software factory cost to run?#
Token price is the wrong number to plan with. In a real coding session, cache reads are the overwhelming majority of billed input: Anthropic's documentation shows a session at "91% of input tokens from cache", and an independent measurement put cache traffic at roughly 80% of the bill. Cache hit rate belongs on a dashboard, because the invalidation list is exactly what factory designers do casually: switching models mid-session, connecting an MCP server, compaction, or fanning out across worktrees, since the cache is scoped per working directory. Invalidation cascades downward from the top of the prompt, so tool definitions are the expensive thing to change: adding one MCP server invalidates the tools, the system prompt and the conversation behind them. Long turns can also expire their own cache, because the five-minute lifetime runs from the start of the request that wrote the entry rather than from the end of the response.
Sources: Bai et al., 29 April 2026 for the phase split; Vanamo, 6 July 2026 for the cost-to-merge comparison.
A cheap model does not produce a cheap merge. In that comparison the cheapest coder spent about $0.05 generating and $2.81 to $5.73 in total, because 97% landed on the review gate. Routing helps less than the price spreads suggest, and cascades move risk without removing it. Running open-weight models locally is a real option, and the reasons that hold up are data residency, air-gapped estates and licence terms rather than per-token price. Budget in cost per useful outcome: a model ten times cheaper per token, with a pass rate three times worse, needing four attempts, is not cheaper. For scale at the other end, Cloudflare publishes 30 days of operating figures for its AI review gate: 131,246 runs across 48,095 merge requests, median completion 3 minutes 39 seconds, $1.19 per review, an 85.7% cache hit rate, and engineers overriding the result 0.6% of the time (Cloudflare, 20 April 2026).
How does a software factory learn from its own runs?#
Without a closed loop, a factory repeats its mistakes systematically and fast. The loop that matters is the mechanism by which an outcome changes the context, checks and workflows every subsequent run uses, which is a different thing from a retrospective.
Concretely: an agent hits a failure, repairs it, then reasons about why. Either a principle was not followed and future runs should follow it, or following it consistently produces worse results and the principle itself needs revising. The critical property is that this happens because the system recognised the problem, without a person noticing first. Steve Yegge describes what this loop looks like across dozens of agents in our conversation about his software factory.
What separates it from a memory feature is review. Learning works as a pipeline: an outcome produces an observation, the observation produces a candidate change to context or to a check, someone decides whether it is true and whether it generalises, and the change lands scoped and versioned with an owner. The payoff is that improvement stops being personal. We have seen a shared skill committed to a monorepo followed by a reported doubling of merged pull requests the next day. That is an anecdote and we report it as one; the same channel carries harm at the same speed, which is why the review stage is not optional.
What tools does a software factory need?#
The sections above describe what each station has to do. This is what organisations are running in them in
- Read the names as examples of a category: the layers are durable, the tools turn over every few months.
| Layer | What it has to do | What runs there today |
|---|---|---|
| Intent | Turn a request into acceptance conditions something can check | Spec tooling that produces reviewable requirements, design and task artefacts before code; acceptance criteria as executable tests |
| Context | Serve the smallest high-signal set, versioned and owned | AGENTS.md at the repository root, nearest file winning; MCP servers over a service catalogue; structure-aware retrieval alongside embeddings |
| Execution | Give agent work a blast radius you chose | OS sandboxes (Seatbelt, bubblewrap, Landlock) with default-deny egress; devcontainers; microVMs; ephemeral CI runners and pods |
| Assurance | Catch the classes tests cannot | Property-based testing (Hypothesis, fast-check, proptest); mutation testing (Stryker, PIT, mutmut); SAST and SCA (Semgrep, CodeQL, Trivy); bounded model checking (Kani, CBMC) |
| Governance | Make the mode per class of change enforceable | Branch protection and code owners; policy as code (OPA, Cedar, Conftest); signed provenance (Sigstore, SLSA v1.2) |
| Telemetry | Show cost and behaviour per run | OpenTelemetry GenAI conventions for tokens, cache and tool calls, plus your own cost layer, because they define no cost attribute |
| Learning | Turn outcomes into reviewed changes to context and checks | A versioned context repository with review; private evals built on your own repositories rather than public benchmarks |
Examples current in September 2026 rather than endorsements; every row has maintained alternatives. Each gate should also state its scope: free Semgrep analyses taint within a single function, and cross-file analysis is a paid tier, which is exactly the boundary agent-generated changes cross.
Two patterns recur in published accounts of organisations running this at scale, and both differ from the
how-to literature. Context comes from the developer portal rather than a vector store. Cloudflare exposes
a catalogue of 2,055 services to agents over MCP alongside AGENTS.md files generated across roughly
3,900 repositories
(Cloudflare, 20 April 2026);
Spotify serves
ownership, documentation and team channels the same way. Parallelism runs on ephemeral infrastructure.
Spotify schedules its background agent as Kubernetes pods; Cloudflare runs up to seven specialised
reviewers as CI jobs with a coordinator that deduplicates and judges their findings. We found no published
account from a named engineering organisation running agent fleets on worktrees, which dominate the
tutorials.
What is comprehension engineering?#
The asymmetry is simple. The time to build software keeps falling; the time for a human to understand it does not, because the human has not changed. The rhythm changes too: you used to think for two hours and work for two weeks, and now an agent returns every twenty minutes demanding the next decision, continuously, from several at once. That structure predicts strain, and we see it in delivery engagements, though the evidence does not yet support the stronger claim that it causes burnout.
Context engineering keeps the machine ready to act. Comprehension engineering keeps the human ready to decide.
— Pini Reznik, From Copilot to the Dark Factory, Dev Summit 2026
Everything so far has been about making machines ready to act. Comprehension engineering is the other half, barely engineered anywhere we have looked: the practice of designing systems, artefacts and feedback loops that keep human understanding of AI-mediated work sufficient for the judgement being asked. The load-bearing word is sufficient, and it gives the idea a scaling variable. The explanation a person needs is proportional to the impact of the decision they face, so a tiny diff may carry high decision impact while a huge generated change carries almost none.
Peter Naur established why this happens in 1985. The real product of programming is the theory in the programmers' heads; the text is its residue. That theory was built by building, so when an agent builds, the by-product stops arriving. Detachment is then forced on people, and re-entry becomes the design question. The accumulated version has a name, and it is not ours: Addy Osmani defines comprehension debt as "the growing gap between how much code exists in your system and how much of it any human being genuinely understands" (14 March 2026).
A better pull-request summary does not fix it. Copilot and CodeRabbit both produce good ones, and they share three limits: they summarise from the branch base and not from the last state a named person understood, they do not know which decision is waiting, and they never check that understanding transferred. The working model has three parts, each an obligation on the system. The decision scopes everything else. The delta is semantic: "843 lines changed" is a diff, "the authentication trust boundary moved to the shared identity service" is a delta. The check is a short self-test before a high-impact decision, and if it fails, the first assumption is that the system failed to transfer understanding.
None of this argues that humans should review more. It resolves into two levers pulled together: fewer gates, through context, validation, deterministic checks and rollback; and better gates, where judgement is genuinely required. The case against belongs here too. Explanation is known to backfire: Bansal et al. found that "explanations increased the chance that humans will accept the AI's recommendation, regardless of its correctness". So the claim is not that briefing improves decisions, but that it lowers the cost of verification. Calling this a discipline would be premature: it is an argument with a design behind it, and no measurements, including from us.
The line to hold on to
A dark factory cannot afford dark humans. The darkness describes the lights on the factory floor; the people are still there, at the ends of the process. Fewer human gates leaves the remaining gates carrying more of the accountability, which is why they have to be built deliberately.
How are teams organised in a software factory?#
The shape this points at is a hybrid team: stable human accountability, with machine capacity that grows, shrinks and changes shape as the work changes. Sized by judgment load rather than work volume, because volume is what the machines absorb.
The working unit is a team of people and machine actors organised around an outcome. Its stable shape is not known. What does hold is that machines take the middle of the work (search, generation, execution, repetition, testing, inspection) while humans hold the edges: intent, judgement, architecture, responsibility, acceptance.
Two arguments about seniority are worth holding together. Relying on senior engineers to supervise agents may be temporary, because their advantage is organisation-specific knowledge that stops being scarce once it is codified as context. Against that, as routine work is absorbed, what remains at a human gate is disproportionately the hard and irreversible part. Either way, if a team shrinks to a few seniors directing agents, the entry-level rung disappears and nobody comes up behind them. Human parallelism is bounded too: reducing what a person needs on return is what raises the number they can supervise, and practitioner reports run from one significant change at a time to three or four. Say several agents, and not many.
Can a software factory work on legacy code?#
On legacy, the objection that this only works for greenfield is wrong; the distinction is quality. Agents cannot work on what they cannot run, so build reconstruction is its own funded workstream: in one benchmark the best configuration got 6.69% of Python repositories running. Characterisation tests pin behaviour without establishing correctness, so an agent writing them will faithfully pin existing defects. And remediation happens only if the organisation funds the time; asking developers to repair a test suite in their own hours works for about a month.
Should you build a dark factory?#
A dark factory is a delivery system where routine implementation and validation proceed without a person writing or reviewing each change. People are at the ends, at intent and acceptance, and the middle is autonomous with interrupts where they matter.
We place this on the AI Adoption Ladder: nine stages, each defined by the bottleneck blocking the next, with the Lights-on Factory at 6 and the Dark Factory at 7. Most organisations we assess sit around stage 2, where adoption is wide and the dashboards still look healthy. The ladder measures adoption; the eight states above measure where the constraint sits.
The more useful half is where not to climb. Internal utility software can stop at 2 or 3; a production product might target 4 or 5; a regulated platform might reach 6 or 7 and keep a human release decision permanently. Different repositories in one organisation legitimately sit on different rungs. Public examples show the range: Stripe's Minions merge over a thousand pull requests a week with no human-written code, all reviewed by people, while Uber reports over 70% of pull requests coming from agents. Read them for mechanism; output volume establishes neither autonomy nor return.
The measurement that does bear on autonomy is consistency across repeated runs. τ-bench defines pass^k as "the chance that all k i.i.d. task trials are successful", against pass@k's chance that at least one is, and found an agent averaging above 60% success dropping below 25% at pass^8. Pass@k answers whether a model can do the thing at all. Pass^k answers whether you can leave it running, which is the question a rung on the ladder actually asks.
Where should you start with a software factory?#
Start with one real delivery workflow that has a measured baseline and explicit acceptance conditions. The first objective is a complete, observable path from requirement to accepted change.
- Time the value stream before choosing anything. It tells you whether the constraint is implementation, unclear intent, integration or release, and it usually changes the plan.
- Do both tracks at once. Working CI, usable documentation and tests that mean something become critical exactly when agents arrive; shared prompts, skills and guardrails make the agents consistent. Doing only the second leaves the first in the way.
- Choose the team with the best backlog discipline, then a bounded vertical slice from its real backlog: big enough to expose real dependencies, small enough to diagnose failures. Establish environments, permissions and protected checks before granting execution authority.
- Develop the engineers on a parallel track. Spaced weekly practice on real backlog work, cohorts that learn together, internal champions trained to teach, as in Why We Teach Agentic Coding Backwards.
On measurement, most of what is easy to collect measures nothing that matters: agent counts, tokens, prompts and pull requests are diagnostic at best. Four questions are worth answering. Are changes reaching users sooner, as lead time from accepted requirement to production including queues and rework? Is the output acceptable, as acceptance rate and escaped defects on comparable work? Is the system economical, as cost per useful outcome including human effort? And where did the constraint move since the last review? That last answer should keep changing, and that is the signal the programme is working.