A SQL query can run without errors, return a complete set of results, and still answer the wrong question.
I saw this firsthand at Turing, where I worked on text-to-SQL models. We could generate a query that passed every syntax check and executed successfully against the database. But when we compared the results with what the user had asked, we sometimes found errors in how the question had been interpreted.
Syntax errors are straightforward to investigate. The database tells you what failed, and you can start looking for the cause. Incorrect results are more difficult to catch, especially when the query is valid and the returned data appears consistent. You need a way to verify that the results match the original request.
I've worked on similar problems throughout my career, from data pipelines to LLM applications. It's also closely related to what I'm doing today at re:cinq, where I joined in September as a Lead AI Native Engineer.
Tracing errors through data pipelines#
I studied Computer Science at LUMS in Lahore before moving to Germany for a master's in Data Engineering and Analytics at TUM in Munich. Since then, I've worked across software engineering, data platforms, and LLM systems.
At ibex, I built analytics for car manufacturers' dealer networks. We collected daily feeds from around six systems, processed the data, and brought it together into reports. Later, at PartnerLinQ, I managed around 80 data pipelines and helped replace manual file exchanges between trading partners with automated integrations.
Working with multiple data sources meant dealing with different formats, dependencies, and expectations. An issue in one part of a pipeline could affect reports produced further downstream, even when every scheduled job had completed successfully.
We relied on tests, audit trails, and data lineage to identify problems and trace them back to their source. If someone questioned a number in a report, we needed to understand how it was calculated, where the data came from, and whether it had been processed correctly.
Many of those concerns became relevant again when I started working with language models, although the methods for evaluating their output were quite different.
Evaluating text-to-SQL and LLM answers#
With LLMs, incorrect answers can be harder to detect because the response can appear perfectly valid. This is a familiar challenge with LLM hallucination, but it becomes particularly important when AI is used to retrieve information or support decisions.
At Turing, we worked extensively on text-to-SQL models, which translate user questions into database queries. Generating valid SQL was relatively easy to test. The harder question was whether the generated query retrieved the correct information.
We developed an LLM evaluation framework that checked the results of generated queries against expected outputs. This helped us detect errors in joins, filters, and question interpretation that wouldn't be caught by syntax checks or successful execution alone.
Later, at Unique, I worked on a GenAI platform for financial services and insurance companies. Accuracy remained important, but we also had to manage document permissions and ensure users could verify the sources behind an answer.
Grounding responses in accessible documents helped users check the information themselves. At the same time, access controls ensured that the model could only retrieve documents the user was authorised to see.
Both experiences showed me how much of AI engineering happens around the model itself. Evaluation, access controls, source verification, and failure handling all need to work together, particularly when AI systems interact with business data.
Building an AI support agent on Lore#
I joined re:cinq because I was interested in applying this experience to software development with AI agents.
We're building software factories, where agents take on a significant share of implementation work while engineers define requirements, evaluate results, and improve the development process.
My current project involves developing an AI support agent for a client's booking system using Lore, re:cinq's agent platform. The agent answers customer questions about their bookings by retrieving information from the client's systems.
A significant part of the engineering work involves managing data access, tool interactions, and failure handling. Customers must only be able to access their own bookings, personal information must stay out of application logs, and conversations need to be transferred to a person when the agent cannot complete a request.
We're implementing these requirements through access controls, guardrails, and a human in the loop for situations requiring manual support.
Unlike a conventional application with predefined flows, an agent can select tools and determine its next action based on the conversation. This creates a wider range of possible workflows and use cases to consider during development and testing.
We therefore evaluate the agent across complete conversations, including missing information, failed tool calls, and requests outside its permissions. The goal is to understand how the system behaves under different conditions and make sure it follows the requirements we've defined.
How spec-driven development changes my role#
We're developing this project using a spec-first approach based on spec-driven development. We define the expected behaviour, requirements, and constraints, and agents implement significant parts of the code.
My role now involves more work on specifications, reviewing generated implementations, and checking whether the results meet the requirements.
Agents can produce code quickly, but that doesn't necessarily mean the entire development process becomes faster. Generated implementations still need to be reviewed, tested, and sometimes corrected. If the volume of generated code increases without improving how we validate it, engineering teams can end up spending more time reviewing than they save during implementation.
This is where the quality of specifications and evaluation becomes particularly important. When generated code passes its tests but doesn't meet the intended requirements, we need to understand what went wrong and how to prevent it from happening again.
Sometimes that means clarifying the specification. In other situations, we introduce additional tests or validation steps. Each issue gives us an opportunity to improve the factory's process, so similar problems can be identified earlier.
Over time, we want more of these checks to run automatically. Engineers will still need to make decisions that involve architectural trade-offs, business context, and requirements that cannot be fully expressed through automated tests. But repetitive validation work can increasingly become part of the factory itself.
That's an area I'm particularly interested in. If agents can generate implementations quickly, we need equally effective ways to evaluate the code and keep the development process manageable as the amount of generated work grows.
Why evaluation is where I want to focus#
There's a connection between the different areas I've worked in. In data engineering, we needed to verify that pipelines processed information correctly. At Turing, we evaluated whether generated SQL returned the expected results. At Unique, we focused on source verification and document access. Now, at re:cinq, I'm applying similar principles to software built by AI agents.
As agents take on more implementation work, I expect evaluation to become an increasingly important engineering responsibility. Teams will need better ways to verify generated code, identify problems early, and automate checks without losing visibility into how the software is built.
That's what I want to focus on over the coming years: improving how we define correct behaviour, evaluate agent-generated work, and build those checks into software factories.
I'm particularly interested in how far we can take automated evaluation while keeping engineers involved in decisions that require deeper technical judgment. There's still a lot to figure out, and that's what makes the work interesting to me.
If you're working on similar problems, whether with coding agents, AI support systems, or data pipelines that have become difficult to verify, I'd be interested to hear how you're approaching them.
You can find me on LinkedIn.
Related articles#
- From Forecasting Models to Coding Agents: Sufyaan Jakate, another new Lead AI Native Engineer, on moving from quantitative finance to agentic systems.
- From the Flight Deck to the Software Factory: Jasper on what aviation taught him about responsibility, checklists, and agents.
- From Platform Engineering to Software Factories and Agentic Organizations: Philipp Strube on why platform engineering will shape the agentic organisation.
