Stop Asking It to Judge: Use the LLM to Read, Score in Code

Previously in this series: The Prompt Won't Save You: Three Objections, Three Tests.

Three experiments told me prompt engineering cannot fix LLM résumé screening, and that most of what I had called bias was the model failing to read the same document twice. So I tried building the thing that should work instead.

Where the last article left off

In the previous part I ran the three objections Reddit raised against my original study of 25,500 résumé screenings. The short version, because this article stands on its own:

The model's score does follow the reasoning it writes. I pasted its own most positive and most negative assessment of the same résumé back into it. If the number were settled first and the text written to justify it, swapping the text would change nothing. The score moved 3.62 points out of a possible 9, 319 times out of 320.

But the model writes new reasoning every single run, and no prompt technique I tested changes that. Reordering the output so the score comes last, the fix most people recommend, moved the run-to-run variation by 0.003 points and made the yes-or-no answer less repeatable.

The reorder does work as advertised. It breaks the anchor, dropping the link between the number and the justification's tone from 0.562 to 0.176. It just leaves the number no better grounded in the model's stated reasons, 0.842 against 0.837, so all it achieves is letting the recommendation follow reasoning that is rewritten every run.

And a control I should have run first showed that most of the movement is not about the candidate at all. Adding one blank line to my résumé moves the score by 0.262 points. Changing my university moves it by 0.230.

That leaves one idea. If the trouble is that the model is making a judgement, stop asking it for one. Use it only to read the résumé and copy out the facts. Then work out the score in ordinary code that a human wrote and can be held to.

This article is that experiment.

The extraction lab

The transplant showed the model's opinion drives its score. Take the scoring away and that stops mattering, because the model is no longer producing a score for its opinion to affect. This is the architecture every engineer I have described the problem to proposes within about ninety seconds, and I had never tested it. The full run is browsable in the extraction lab.

Here is how the test runs.

1. Ask the model to copy the résumé into a fixed set of fields. Employers, dates, titles, degrees, technologies. No score, and no opinion of the candidate. The schema and the exact prompt are in the repository.
2. Never show it the job description. This matters more than it sounds. If the model knows you are hiring a backend engineer, then every choice about what to copy becomes a choice about what is relevant to that job, and judging relevance is exactly what I am trying to take away from it.
3. Parse the identical document five times. Anything that comes back different changed for no reason, which sets the noise floor.
4. Parse all 30 résumé variants the same way, then compare each against the baseline parse.
5. Work out every comparison, total and threshold afterwards, in code.

That is 2,700 parses across all eleven models, at temperature 0 and again at 0.7.

Result 1: temperature 0 is not determinism

Two numbers per model, and both need a word of explanation because I made them up for this experiment.

A parse of my résumé comes out at 296 separate fields. Agreement is the share of those that come back identical when you read the same document twice, so 1.000 means the parse never moved. Leakage is how many fields move when one demographic line changes, after subtracting however many move anyway between two readings of the same document. That subtraction is the whole point. If a model already disagrees with itself on 8 fields, then 8 fields of difference after a name swap tells you nothing.

Both are measured at temperature 0, the setting that is supposed to make a model give the same answer every time. When it does not, the usual explanation is not hidden randomness in the sampling. It is that these models run on GPUs in batches, floating-point addition is not associative, and the same request batched alongside different neighbours can take a slightly different numerical path to a slightly different token. I did not measure the cause here, only the effect. Every row below is browsable, model by model, on the extraction lab page.

One model out of eleven parses the same document identically every time. Everyone else disagrees with themselves on between 2% and 28% of extracted fields, reading the same text twice, with the temperature set to zero.

Result 2: perfectly consistent, and still rereading the career

This is the one I would publish if I could only publish one.

Look at the top row again. Gemini 2.5 Flash reads the same document the same way every single time. Parse it five times, get five identical answers, right down to the last field. And it has the highest leakage in the set.

I changed only the first name at the top of the résumé. Nineteen fields moved. You can pull up that exact parse and read it field by field against the baseline.

I checked the documents byte by byte. The employment section of those two files is identical. Flash returns the same fifteen roles, same employers, same titles, same dates, in the same order, and then reclassifies what the person actually did.

employment[1].functions        {backend, data}     -> {fullstack}
employment[2].seniority_level  "founder"           -> "c_level"
employment[4].functions        {backend, data, distributed_systems} -> {backend, data}
employment[8].functions        {fullstack, mobile} -> {fullstack}

Every axis does this. Change the address country and 20.0 fields move. Change the school, 22.8. Company locations, 26.3. Graduation year, 19.5. And remember this model has no random variation at all, so none of that can be chance.

This is cleaner evidence than anything in my original study, and I want to be precise about why. In the first article, every result had an escape hatch. Maybe the score only moved because these models are a bit random, and a rerun would have moved it back. That was a fair objection, and only about 4% of the score shifts were big enough to rule it out. I said so at the time.

Here there is no escape hatch. This model does not vary. Ask it the same question a hundred times and you get the same answer a hundred times. So when the answer changes, something in the input changed it, and the only thing I changed was the name.

It also turns the usual advice about temperature on its head. At 0.7, Flash starts disagreeing with itself on about 21 fields per run anyway, and the 21 fields that move on a name swap no longer stand out from that. The bias has not gone anywhere. It is now hidden inside the randomness. Turning the temperature down does not make a model fairer, it just stops the randomness covering for it.

Scoring it in code

I built the other half, a deterministic scoring function over the extracted data.

The scorer reads only the factual tier. Technology credit comes from searching the source document, never from the model's claim about it. Experience is recomputed as a union of overlapping intervals, because my résumé has five concurrent roles and naively summing durations turns a 20-year career into 38. The job requirements live in a versioned config file, seventeen weighted requirements and one disqualifier, and those weights are a policy choice, not a measurement, which is precisely why they belong in a file a human can read.

Then I scored all 2,700 extractions. The full results are in scored.json, one row per model with the score change for every variant.

Does it remove the bias

For nine of the eleven models, the score moves on exactly one variant out of 29, and that one is legitimate. The fully-anonymised variant scrubs the university, so the degree requirement genuinely cannot be verified. The document really did lose the information.

Name, country, school, employer, company location, graduation year, career gap. Delta exactly zero.

That includes Gemini 2.5 Flash. The same model that reclassifies nineteen fields when you change the name scores the identical number regardless of it.

But not for the reason people expect. This does not make the model fair. The bias is still sitting right there in the data it copied out, exactly as measurable as before.

Here is what that looks like, using Gemini 2.5 Flash because it reads the same document identically every time, so nothing below can be chance. Change my address from Berlin to Lagos. That edit touches one line of the document and nothing else.

MCP is named once in each document, in a paragraph the edit does not go near. The model just stopped recording it. My distributed systems work disappears the same way.

The seniority label on my founder role moves too, and it moves in whichever direction the edit pushes. c_level when the first name is Aisha Okonkwo. staff when the address is Lagos. staff again when the university is MIT. Same role, same dates, same description, three different answers.

None of that is formatting or ordering. Those are the values themselves, and every one of them is a field a scorer could plausibly read. What changes is that the fields it got wrong are not the fields the score is built from. Technology credit comes from a grep. Experience comes from arithmetic. Thresholds get compared in code. The model's opinions, which domains this person works in, how senior they seem, whether their employers look prestigious, go to a human reviewer and are never priced.

What makes it work is that the scoring function never reads those fields.

Four caveats, and they are not footnotes

The insulation is partly luck, and this is the caveat I would worry about most. My scoring function does read the role-function field, the exact field Flash reclassifies. The job spec asks for six years of full-stack work, which is 72 months. Across the variants that do not touch a single date, the months my scorer computes from Flash's own parses range from 151 to 227, a spread of 76 months. That is wider than the threshold itself.

It did not matter for me, because my baseline lands at 222 months and nothing in that range comes near 72. It would matter enormously for a candidate sitting at seven years. The same drift that is invisible on my résumé would push them over or under the bar depending on their name, and the scorer would return a clean, auditable, deterministic answer either way. A threshold does not remove the parser's instability, it just decides who is unlucky enough to be standing on it.

A broken parser cannot be rescued. Qwen's score still moves on 21 of the 29 variants. Of the fields it was asked to fill in, it leaves 13% empty, and what it leaves empty is telling. It routinely returns a job with the opinion field answered and the employer, title and dates all missing. It also never noticed the two-year gap I deliberately inserted into two of the variants, which is the check I built specifically to catch a parser that is not really reading. This architecture protects you from a good parser's bias. It cannot save a bad parse.

A deterministic scorer inherits the parser's instability. Claude Sonnet scored my baseline résumé 8.9, 8.9, 8.9, 8.4, 8.9 across five identical runs. The scoring function is deterministic. The thing it consumes is not.

The last one is the largest, and I would want it on any vendor's slide. Determinism buys auditability, not fairness. A function that weights years-since-graduation is legibly age-discriminatory. Proxy variables survive this architecture completely intact, country of degree, employer location, gap length. What you gain is that the bias now sits in a config file you can read, diff, review and argue about, instead of inside a paragraph of extremely fluent prose.

What I would tell you if you are shipping this

Prompt engineering is not a bias control. I tested the five most obvious techniques and none of them moved demographic bias at all. If a vendor's fairness story is that they engineered the prompt carefully, they do not have one.

Demand repeatability on identical input, not a benchmark score. Run it five times on the same résumé and look at the spread. That single number tells you more than any eval suite.

Measure the score and the decision separately. Score-last held the number still and broke the recommendation.

If an LLM is in your pipeline, use it to read, not to judge. It works, and it forces you to write your criteria down, which is uncomfortable, which is the point.

Under the EU AI Act, a screening system whose recommendation is not reproducible across identical runs is not an auditable system, however articulate its explanations are.

Limitations

u/Yteburk, an MSc student in Responsible AI, commented on the original thread that they were not sure this was the right way to test it. They never got a proper reply, so here is one. Every metric in this article is defined in code, and every figure in it is recomputed from the raw runs by a script that fails if a number drifts. That does not make the design right, it makes it checkable, which is the most I can offer.

There is one base résumé and one job spec for the scorer, and a one-variable-at-a-time design that cannot see interaction effects. Models change, so all of this is tied to the versions tested. The earlier parts of this series carry their own limitations, which still apply.

The comparison against human recruiters, which several commenters asked for and which I answer in the previous part, still stands unmade. Nothing here tells you whether a person would have done better.

Where this leaves the series

This is the last article in the series, so let me say plainly what four of them add up to.

When we started, Pini Reznik, one of the founders of re:cinq, said something I have quoted before and will quote again. "It is not about models being biased or not. It is about awareness." At the time I took it as a way of avoiding the question. Forty-seven thousand model calls later I think it was the thesis.

The first article established that changing an irrelevant detail moves the score, and that the movement is mostly small, mostly hidden inside the random variation, and almost always downward. The second checked whether the frontier escapes it and found Claude Fable 5 sitting comfortably in the middle of the pack, so capability does not buy fairness and does not cost it either.

The third went looking for a fix, found that prompt engineering is not one, and then found something worse. Against a placebo floor of meaningless edits, only first names and career gaps clearly stand out. My university does not.

This one found the only thing that worked, and it works by keeping the model's judgement out of the decision entirely.

If I could send one thing back to myself in May, it would not be any of the four experiments in this series. It would be the blank line.

None of that adds up to "don't use these models." It adds up to two things. The first is that how well a model explains itself tells you nothing about whether it is right. Every one of these systems will give you a fluent, professional, entirely believable account of why it scored someone a 4, and that account will not tell you whether it would have said 7 with a different name on the page.

The second is that you cannot tell whether a number is meaningful until you know what a meaningless change does to it. That is not a machine learning insight, it is just how measurement works, and I published 25,500 evaluations before I applied it to my own.

If you take one number from this series, take this one. In 1,870 written justifications, not one model mentioned the weather, and every one of them scored the résumé differently depending on it.

What I would hand to whoever picks this up next. A multi-pass extraction pipeline, still untested. More than one base résumé, which is now clearly the biggest limitation of the whole project. Placebo coverage for the Claude models under the same conditions as their demographic runs, which the harness problem prevented this time. And the human baseline u/pjdoland asked for, which is a harder study than this one and probably a more important one.

The data is all open, both as an interactive app and as raw downloads. If any of that appeals, take it.

My thanks to u/kamilc86, u/AssiduousLayabout and u/hex4def6, who between them cost me about 17,600 model calls and roughly six weeks. hex4def6's paragraph was four sentences long and cost me my headline. Best review I have ever had.

Sources

The study

• Hiring Bias web app, every evaluation in the project
• Study repository and raw runs

This experiment

• Extraction lab, all 2,700 parses across eleven models, field by field against the baseline
• Extraction schema and prompt, agreement and leakage metrics and the deterministic scorer
• Versioned job spec, the seventeen weighted requirements and one disqualifier
• Scored results, one row per model with the score change for every variant
• Résumé diffs down to the character

Regulation

• EU AI Act, Regulation (EU) 2024/1689

Read the rest of the series: The Blind Spot in the Machine: What 25,500 LLM Evaluations Reveal About AI Hiring Bias, The Frontier Isn't Special: Claude Fable 5 and Hiring Bias and The Prompt Won't Save You: Three Objections, Three Tests.

Explore the open data and reproduce every figure at the Hiring Bias web app and the study repository. The transplant and prompt-lab figures cover the ten models common to those experiments. The extraction lab covers all eleven, including Claude Fable 5. Prompt-lab stability figures are computed on the *47 of 80 baseline cells with non-zero variance, since 33 cells score identically on every run and cannot be improved upon. All figures are recomputed against the current dataset and differ slightly from the May 29* publication.