Sam Newman on Software Factories: Why AI Can't Fix Your Production Incidents (Yet)

Sam Newman joins Benedikt Stemmildt to talk about resilience engineering, distributed systems, and what changes when AI starts taking on more of the work involved in building software.

The conversation starts with a simple idea from Sam's new book, Building Resilient Distributed Systems: resilience is not about preventing every failure. In a complex system, things will fail. The job is to make sure the system can keep working when they do, recover quickly afterwards, and adapt when something happens that nobody planned for.

From there, they get into what that means for AI. Model providers still fail in ways we would not accept from most managed cloud services. Model behaviour can change. Non-determinism makes testing harder. AI can already help engineers make sense of incidents, but Sam is much more cautious about letting it fix production systems on its own.

They also explore software factories as systems in their own right. If a software factory is responsible for producing software, it needs the same things we expect from any production system: observability, clear boundaries, validation, and a way to recover when something goes wrong. The harder question is how much knowledge about resilience the factory itself needs in order to build software that can survive production.

The episode closes on the organisational side of AI adoption. Sam argues against treating AI as a top-down mandate or assuming every team and every part of a codebase should use it in the same way. Teams need room to experiment, measure what is actually improving, and protect the shared understanding of the systems they are building.

• Resilience is not about making sure nothing ever breaks. It is about keeping the system working when something does.
• In a large distributed system, failures are normal. The design has to account for that rather than pretending they can all be eliminated.
• LLM providers still do not offer the reliability most teams expect from a normal managed cloud service, so fallback options and experience with more than one model or provider are worth building early.
• AI can already help during incidents by gathering signals, finding related changes, and helping build a timeline. Fully automated remediation is a much harder problem.
• A software factory is itself a production system. If teams depend on it to ship software, it needs observability and resilience too.
• Sam recommends starting small with software factories: let them build less critical modules behind clear boundaries rather than trying to generate an entire system in one shot.
• Faster software delivery is only an improvement if teams still understand what they are building. AI should reduce repetitive work, not create more context switching and cognitive debt.

Sam's view of resilience starts with a distinction that changes how you design systems. Traditional safety thinking often focuses on preventing things from going wrong. Resilience engineering assumes that, in a sufficiently complex system, something will eventually fail and asks how the system can keep doing its job anyway.

He uses a simple example. If one component has a one-in-a-hundred chance of failing on any given day, improving that component may be the obvious thing to do. Once you have a hundred components, though, the chance that at least one of them is having a bad day becomes much higher. At that point, trying to make every component individually perfect is not enough. The system needs to be able to keep operating when one of them fails.

Google's early server racks are a good physical example of that thinking. Power supplies and hard drives, the parts most likely to fail, were mounted so they could be replaced quickly. The software was designed not to care too much when an individual machine disappeared, and the hardware was set up so somebody could swap the broken part and get it running again.

Sam connects this to two of David Woods' concepts from resilience engineering: robustness and rebound. Robustness is the ability to keep operating when something goes wrong. Rebound is the ability to recover afterwards. You need both, and the balance changes depending on the system you are building.

AI adds a new set of reliability problems because the models themselves are not stable in the same way as traditional software components. Foundation model providers can have availability problems, model quality can change without much warning, and the same input can produce different outputs.

That makes observability and evals much more important. If an AI-enabled system is going to sit in production, teams need ways to see what it is doing and to check whether its behaviour is still within an acceptable range.

Sam also recommends avoiding total dependence on a single model provider. He is not suggesting that every request should be sent to several providers at once. A more practical approach is to run part of an agentic workflow on a different model or provider. That gives the team some real experience with alternatives before they are forced to switch.

Where Sam is more optimistic is incident response. AI can already help engineers collect information, connect signals, surface recent code changes, and build an initial timeline for a post-incident review. Those are all places where reducing the amount of manual searching can help.

Automated remediation is different. Incident response is rarely a neat, linear process, and even deterministic remediation is difficult to get right. Letting a non-deterministic model decide how to repair a production system introduces a much higher level of risk. For now, Sam sees AI as far more valuable in helping people understand an incident than in letting it take over the response.

The conversation then moves from AI inside production systems to AI being used to build those systems.

Sam describes distributed systems as socio-technical systems: the technology and the people operating it are part of the same system. A software factory should be treated the same way. If teams depend on it to produce software, it is not just a developer tool. It becomes part of the production path and needs to be designed accordingly.

The comparison with continuous integration is straightforward. If CI is down and nobody can ship software, CI is a production system. A software factory is no different. Teams need observability into how it works, what it is producing, and where it is failing.

The more difficult problem is what the factory needs to know about the software it creates. A production service may have an SLO for availability or latency, but we do not yet have a reliable way to hand those requirements to a software factory and expect it to design the right system by itself.

Sam's suggestion is to avoid jumping straight to the most extreme version of the idea. Physical dark factories usually produce components rather than entire finished products, and software factories can be approached in a similar way. Start with a few modules where the risk is manageable, put clear boundaries around them, and keep people responsible for the wider system design and how the pieces fit together.

That gives teams a way to learn where software factories work well without betting the whole architecture on them.

The last part of the conversation is less about models and more about how organisations adopt them.

Sam prefers teams that own a business domain end to end and can decide for themselves where AI or a software factory actually helps. In Team Topologies terms, those stream-aligned teams can then be supported by enabling teams and platform capabilities that make new ways of working easier to adopt.

That is very different from a company-wide mandate telling every team to use AI in the same way. Teams work on different kinds of software, with different levels of risk and different needs. Treating all of that work as interchangeable makes it easy to optimise for activity rather than outcomes.

Sam argues that organisations should measure both sides. Are teams actually delivering more? Are users happier with what is being shipped? Are incidents increasing? Are people working longer hours or burning out? More generated code is not automatically a sign that the system is improving.

He uses Jennifer Cox's phrase "ship fast enough to care" to make the point. Going faster is only valuable while teams still have enough time to care about the product, the people building it, and the people using it.

That connects to cognitive debt. If AI makes it easy for everyone to work in parallel and produce more code, teams can gradually lose their shared understanding of the system. Sam brings in Peter Naur's idea of programming as theory building: software is not only the code in the repository, but also the mental model shared by the people who understand why the system is the way it is.

AI should give teams more time for that kind of thinking, not less. If it removes repetitive work but replaces it with more context switching, more fragmented work, and less conversation, the team may be moving faster while understanding less.