Evals, reducing hallucinations, & AI-native development

On This Podcast

In this conversation, we examine how TESSL's documentation registry grounds coding agents in accurate, version-matched documentation for open source libraries, cutting down on the API hallucinations that plague any code newer or older than a model's training cutoff. Amy explains that a tessl.json file ties each project to the exact package versions it depends on, so agents read optimised summaries instead of burning tokens re-deriving first principles from node_modules or a live web search every time. We also discuss how the same registry mechanism extends to style guides, internal repo documentation, and spec-driven development instructions, letting teams share machine-readable context across repos in a way Git alone handles awkwardly. Daniel notes the particular frustration of an agent confidently inventing a method that sounds plausible but simply does not exist, and Amy frames the registry as removing an inefficient, repeated search that every agent would otherwise have to perform alone.

In this conversation, we examine how building evaluation harnesses for coding agents pulls software engineers into unfamiliar territory, closer to a biological scientist running experiments than a developer chasing a deterministic pass or fail. Amy describes constructing baskets of synthetic scenarios, running an agent against each one multiple times to capture variance, and grading the results to see how a change in documentation, model, or agent shifts the average score rather than fixing a single broken test. We also discuss how this statistical mindset mirrors product management, where success is read from metrics trending in a direction rather than a single provable truth, and Daniel wonders aloud whether software engineers have simply been unusually lucky to work with certainty for as long as they have. Amy points to newer tooling, from Terminal-Bench benchmarks to simulated-user harnesses like the one she built internally, as the field's early attempts to standardise evaluation as agents take on longer, multi-turn tasks.

In this conversation, we examine a pattern Amy's team keeps finding in enterprise codebases: agents.md and cursor-rule files that accumulate contradictory or outdated instructions over time, leaving an agent to silently decide which rule to obey, which one of Claude's skill descriptions functions less like documentation and more like a pitch persuading the agent it is worth reading at all. We also discuss the paradox that hyper-detailed task prompts can cause a model to ignore its broader system instructions altogether, and Amy recounts a model flatly refusing to use a newer version identifier because it contradicted its own training. Daniel connects this to the Van Halen brown M&Ms story as an informal way to detect when an agent's context is overloaded, and both agree the fix is architectural rather than cosmetic: splitting a task's competing concerns, security, performance, readability, correctness, into separate passes or separate agents with a clean context, rather than asking one model to juggle every constraint in a single conversation.