The Prompt Won't Save You: Three Objections, Three Tests

Previously in this series: The Frontier Isn't Special: Claude Fable 5 and Hiring Bias.

I posted my hiring-bias study to Reddit. Three commenters told me I had measured the wrong thing. I ran all three of their objections as experiments, and the last one cost me my headline.

The comment I couldn't argue with

In May I published a study of 25,500 LLM résumé screenings and posted it to r/artificial. One base résumé, my own, one changed detail at a time, eleven models, seventeen job descriptions. An independent auditor flagged roughly 45% of the score differences as bias-driven.

I expected the usual split, people who found it alarming and people who found it obvious. I got both. I also got three commenters who went after the method, and their objections were sharper than the post they were attacking. One of them eventually cost me the headline.

The first came from u/kamilc86, and it went straight at the weakest part of the study.

That 45% is mostly instability getting labeled as bias. If swapping one field moves the score, the model has no stable scoring function, and that alone disqualifies it for screening even if every shift were demographically neutral.

The invented justification is just the model rationalizing a score it already produced, so the explanation tells you nothing about what moved it.

That second paragraph was the problem. My whole article was built on quoting the models' own reasoning, especially the moment where Gemini praised my geographic-mapping work on the baseline and then, with an MIT degree pasted in, described the very same experience as "not directly related." I had presented that as how the bias works. Kamil was saying it was only for show. If he was right, every quote I published was decoration and I had been seeing a pattern that was not there.

His first paragraph turned out to matter even more, but it took me two experiments, plus a control I had not thought to run, before I understood why.

The second came from u/AssiduousLayabout, who had actually read my prompt, which is more than most people do, and found the problem I had stopped noticing.

A key mistake that I see here is that you list the score first in the output object. What this causes the LLM to do is first pick a score and then retroactively justify that particular choice of token... I would bet decent money the following will produce far more stable results.

Then they wrote out the corrected prompt. Strengths, concerns, key factors, justification, recommendation, and the score dead last. u/Nalmyth summed up the reaction, mine included: "this user above you just wrote a paper in a Reddit comment, makes very clear logical sense, and oh now I have to go reorder a few things."

Both objections could be checked, and that is rare. Most criticism of a study is a feeling, and you can agree politely and move on. These two each described a test I could run, and I already had everything needed to run them.

So I ran them.

Experiment one, the reasoning transplant

Kamil's claim has a clean test. If the model picks a number first and then writes an explanation to fit it, then handing it a different explanation should change nothing. The number was already decided. But if the model works out what it thinks and the number comes from that, then changing the explanation should change the number.

So I built what I started calling a reasoning transplant, and you can walk through every cell of it. This one is Kamil's experiment. He wrote the hypothesis, I only ran it.

Take one résumé and one job. Ask the model to assess it several times, writing strengths, concerns and a justification, but no score. Now take the two extremes it produced for that exact document, its most positive assessment and its most negative one. Then score the résumé twice more. Once with its own positive text pasted back in, once with its own negative text. Everything else identical.

If Kamil is right, the effect is zero.

Across 320 cells and ten models, swapping the negative assessment for the positive one moved the score by 3.62 points on a ten-point scale. The score moved in the reasoning's direction in 319 of 320 cells. One Llama cell went the other way, everything else fell in line.

So the objection is wrong. The model writes its opinion first and the number comes out of that opinion. It is not choosing a score and then dressing it up. The quotes I published were showing real workings, not a cover story.

Except Kamil was still partly right. The score follows the reasoning, but it does not follow it all the way, and I can show you what that looks like.

Here are the two assessments Claude Opus wrote about the same résumé for the same CTO job, minutes apart. Each evaluation ends with three factors the model says drove its thinking, each marked positive or negative and weighted high, medium or low. This exact cell is on the site, with both full write-ups and all five scores from each arm.

The positive one:

positive  high    Hands-on agentic AI and MCP framework experience
positive  high    Distributed systems and low-latency backend depth
positive  medium  Fintech and regulated-environment exposure

The negative one:

negative  high    No executive leadership or team-scaling track record
negative  high    Lacks production multi-agent AI systems experience
negative  high    Missing fintech/regulated-environment depth

Read the third line of each. One says the candidate has fintech exposure, the other says the fintech depth is missing. Same document, same model, opposite conclusions about the same fact.

Put a number on the distance between them. Counting a high factor as 3 and a medium as 2, positive up and negative down, the positive assessment lands at +8 and the negative at -9, a 17-point gap out of a possible 18. Across all 320 pairs the average is 14.8 out of that 18, so this pair is typical, not an extreme I picked out.

So the reasoning swings almost the whole way it could swing. What does the score do? It runs from 1 to 10, so it could move at most 9 points. It moves 3.62.

The reasoning goes 82% of its available distance, the score follows 40% of its available distance, so the score moves about half as far as the reasoning does. Hand the model a positive write-up and the number rises, but only halfway to where that write-up points.

The model turns up with a rough score already in mind, and the reasoning pulls it around instead of setting it from scratch. The reasoning has real influence and it is also fighting something. Which is neither what I claimed nor what Kamil claimed.

Put two facts together. The score follows the reasoning. The model writes brand new reasoning every time you run it. So when the same résumé scores 7 on one run and 5 on the next, that is not the number rolling around at random. The model genuinely thought something different the second time, and the score honestly reflects it.

Kamil's first paragraph, the one about there being no stable scoring function, turned out to be the more serious half of his comment. My own experiment had just explained why it happens.

Experiment two, the prompt lab

Now for AssiduousLayabout's reorder. This one I expected to lose.

AssiduousLayabout's logic is sound, it matches how these models work, and I would have bet on it myself. So I built a prompt lab. Six strategies, all producing an identical output schema so only the technique varies. The naive baseline with the score first. Score-last, exactly as specified in the comment. A competency rubric. An explicit blind instruction. Chain-of-thought. Few-shot examples. Four résumés, two jobs, ten models, ten runs each, 4,800 evaluations.

Start with the score itself. Run the same résumé five times through the same prompt and the answers spread out a bit. That spread, measured in score points, is what score-last is supposed to shrink.

One thing has to be handled before the comparison means anything. 33 of the 80 cells already score an identical number on every single repeat run, so their spread is exactly zero and no prompt can improve on them, only tie or lose. Including them would make whichever prompt you started from look better than it is. On the 47 cells that had room to move, score-last comes in at -0.003 points (t = -0.06).

Not worse, not better. No change at all. The most-recommended prompt fix on the internet, applied exactly as written, changed the run-to-run movement by three thousandths of a point.

But the bet is not lost yet, because there is a second thing to measure. Forget the number. Does the model give the same yes, no or maybe across identical repeat runs?

There are 80 of these cells, each one a résumé and job scored five times by one model. Under the naive prompt, 33% of them come back split rather than unanimous. Under score-last that rises to 54%. Counted properly, 25 cells that had been unanimous became split, and only 8 went the other way, which is far too one-sided to be chance (McNemar chi-square 7.76, p about 0.005). You can put the two prompts side by side and step through all 80 cells, or take the raw runs and count them yourself.

So making the model reason before deciding left the score exactly where it was and made the actual hiring decision measurably less repeatable. AssiduousLayabout was right about the mechanism, the score-first anchoring is real, and u/exjackly was right too when they noted that even reasoning models keep the anchor: "even when it recognizes that there is an error, it frequently maintains the same error with continued thinking." Being right about the mechanism did not produce the fix.

Then I checked the thing the study is actually about, whether any of the six strategies reduced demographic bias.

"vs baseline" is that strategy's mean minus the naive prompt's, so negative means less bias. The t column asks whether that difference is larger than the run-to-run variation behind it, comparing each strategy against the baseline on the same 60 combinations of model, job and résumé variant. As a rough rule, a t under about 2 means the difference could easily be noise.

Every value in that column sits between -0.53 and 1.52. Not one strategy separates from the baseline in either direction. The best performer is noise.

The blind instruction deserves singling out, because it is the fix everyone reaches for first. I literally told the model to ignore identity signals. The score change stayed the same within measurement error. And the rate at which a demographic swap flipped the actual yes-or-no answer went up, from 8.3% of comparisons to 11.7%. Telling a model not to be biased does not make it less biased. It makes you feel better.

The two answers are one answer

These are not two findings.

The transplant says the score genuinely follows the written reasoning. The original study showed the model writes new reasoning on every run. The prompt lab shows that no reordering, rubric, instruction, chain-of-thought or exemplar set interrupts that.

Chain them. Identity signal, then narrative, then score.

That is why the prompt cannot fix it. A prompt controls the shape of the reasoning. It does not control the fact that the reasoning gets regenerated from scratch, with different emphasis, every time you run it.

You can tell the model what order to think in. You cannot tell it to think the same thing twice.

Which brings me back to Kamil's first sentence, the one I initially read as criticism and now read as the finding. If swapping one field moves the score, the model has no stable scoring function, and that alone disqualifies it for screening. u/OkParfait4006 made the same point from the other direction: "the 6x stability difference between models is the number worth paying attention to... consistency under slightly different inputs is the metric that actually matters for anything used in real decisions."

They were both describing the conclusion of an experiment nobody had run. I had the data to run it and had not thought to.

Experiment three, the control

Buried further down the thread, u/hex4def6 left a one-paragraph suggestion that I acknowledged politely and then did not act on for weeks.

I think it would also be interesting to include some clearly irrelevant factors (color of the person's car, day of week submitted, weather forecast on the day of resume submission, etc). If those move the needle significantly, it just points to the instability of this test.

This is a placebo control, and it is the thing my study was missing. I had been measuring how much the score moves when I change a name. I had never measured how much it moves when I change something that could not possibly matter. Without that floor, I had no idea whether 0.4 points was a lot.

So I built seven placebo variants. Two mention a car ("Applicant vehicle on file: red Volkswagen Golf", and the same in silver). You can read every one of them against the original résumé, down to the character. Two mention the day the application arrived, two the weather. And one, the important one, adds a single blank line and changes nothing else at all.

Then I ran the whole grid again. Seven placebos, seventeen jobs, five runs, on the seven models I could run identically to the originals. 4,165 more evaluations.

Here is what came back, measured exactly the way the demographic numbers are measured.

Changing the candidate's identity moves the score about 10% more than mentioning their car. The difference is real and it is tiny: paired job by job, 0.034 points, t = 3.03.

And adding one blank line to a document moves the score by 0.262 points. Not a word changed. Not a character of content. A newline.

Broken out and ranked against the demographic axes from the original study, they do not separate into two groups at all. They interleave:

The single largest effect in the whole study is the colour of a car I do not own. The smallest is which university I went to.

Mentioning my car moves the score more than changing my university, my country, my graduation year or my employers. Every one of these numbers is negative on average too, so like the demographic edits, the irrelevant ones mostly push the score down.

The part that actually bothers me

You could argue the car results are unfair. The car line adds text where a name swap only replaces it, so maybe the model is reacting to a longer document rather than to the car.

So compare the two car variants against each other instead. Both add one line, in the same place, of near-identical length. The only difference between them is the word "red" or "silver".

That swap alone moves the score by 0.235 points. The submission day moves it by 0.207, clear weather against rain by 0.207. Gemini 2.5 Flash is the worst of them, changing its score by 0.518 depending on the colour of a car that appears nowhere in the job description and has nothing to do with the work.

So it is not just document length. The model is reading the value and pricing it.

Those are averages across seventeen jobs, which flattens the worst of it. Here are the widest single swings, each one the gap between two variants that differ only in the value of an irrelevant fact:

Telling Gemini 2.5 Flash the application arrived on a Saturday rather than a Tuesday costs the candidate two full points out of ten. Two points is the distance between a clear yes and a clear no on most people's threshold.

Look at the first two rows together. Same job, same fact, two models from the same vendor, and they move two points in opposite directions. Flash prefers Tuesday, Pro prefers Saturday, by exactly the same margin. There is nothing to interpret here. Neither preference means anything.

And the blank line has a worst case too. On the junior fullstack job, adding one empty line to my résumé cost me 1.6 points with Gemini 2.5 Flash.

I want to be careful not to claim too much from a table of worst cases. These are the extremes of 561 paired comparisons and the averages are much smaller, which is why I led with the averages. But the extremes are where the hiring decision actually lives. Nobody is averaged across seventeen jobs. A candidate is one row, applying for one role, on one day of the week.

The last number is the one I keep coming back to. I searched every justification the models wrote under these conditions for any mention of the car, the day or the weather.

The models never bring it up. In 1,870 written justifications, the car is referenced twice and the weather and the day never. They score a silver Golf differently from a red one and then explain their decision entirely in terms of my backend experience.

That is the same silent mechanism I described in the first article, where a model reframed my mapping work as irrelevant rather than saying anything about MIT. I had read that as proof of hidden bias. The control says it is something wider, and less helpful to the story I told. These models are influenced by things they never mention, and a car colour is one of them.

One way to read that table is against the placebo floor of 0.328, the average of all seven meaningless edits. First name, career gap, the anonymised résumé, company locations, company names and graduation year all sit above it. Address country and school sit below it, which means changing which university I went to moves the score less than adding a blank line does. So does changing which country I live in.

I want to be careful about what this does and does not overturn. The individual cases in the first article are still real. Gemini really did drop my score by 2.8 points when I swapped my university for MIT, and it really did reframe the same geographic-mapping work as irrelevant to justify it. What the control shows is that the school axis as a whole does not rise above the noise, so that example was a dramatic instance of something that is not a reliable pattern. I led with it, and I should have had this floor before I did.

Two models, Claude Haiku and Gemini 2.5 Pro, show a placebo effect equal to or larger than their demographic effect. For them, the bias reading in my original study is not separable from the noise this control measures.

What survives is narrower and, I think, more defensible than what I published in May. First names and career gaps clearly move the score more than a meaningless edit does. Most of the other axes do not. And all of them sit inside something much larger, which is a model that cannot read the same document twice the same way.

Which means u/kamilc86's first sentence, the one I spent two experiments working around, was simply correct. That 45% is mostly instability getting labelled as bias. I needed a control to see it, and he did not.

The wrapper is part of the prompt

Almost nobody sends a résumé to a raw model. It goes through an SDK, a framework, an orchestration layer, an agent, or a chat product, and every one of those puts its own text into the context window before your question arrives. That text is not neutral, and the control gave me a way to measure exactly what it costs.

The Claude models here are a worked example. They do not go through a plain API call. They go through the Claude CLI, because that is the subscription I had. The CLI is a coding agent, so it loads roughly 29,000 tokens of system prompt, tool definitions and operating instructions before my résumé prompt arrives. Every Claude score in this project was produced by a model that had just been told it was a software engineering assistant with a file editor, and was then asked to evaluate a candidate.

So I measured it. I re-ran the Claude baselines through a clean invocation with that context stripped out. Same résumé, same seventeen jobs, same five runs, 2,720 evaluations.

Three of the four score the same candidate higher when a coding agent's instructions are sitting above the question. The average is 0.138 points, and for Opus it is 0.247.

Now put that beside what this study measures. Claude Opus's entire demographic effect is 0.282. The context added by the tool I happened to call it through is 0.247, which is 88% the size of the signal itself. The wrapper is nearly as loud as the thing I was trying to hear.

Two things follow, and the second is the one that matters to anyone shipping this.

Within a single model, the comparisons hold. Every variant carried the identical wrapper, so when a name swap moves Opus, the wrapper was present on both sides and cancels out. It does mean the Claude models' absolute scores here run about a tenth of a point high, and it is why the Claude rows are left out of the pooled placebo figures above. Their control runs are clean and their demographic runs are not, so comparing the two would measure my tooling rather than the placebo.

Across models, and across deployments, it does not hold. If you benchmark a model in one wrapper and ship it in another, your numbers do not transfer. A vendor's fairness evaluation run against a bare API says nothing about the same model behind an agent framework, a RAG pipeline that prepends retrieved documents, or a chat product with a long system prompt. Those are all the same intervention as my 29,000 tokens, and my measurement says an intervention that size is worth roughly a quarter of a point.

Every number here is on the control page, with the runs behind them in the downloads.

The irony is not lost on me. I ran an experiment about how much irrelevant context moves these models, through a tool that was adding 29,000 tokens of irrelevant context. If it can hide in a study designed to look for exactly this, it can hide in your pipeline.

What this leaves standing

Three objections, three experiments, and the study I published in May comes out smaller than it went in.

The score does follow the model's written reasoning, so the quotes in the first article were showing real workings. But the model rewrites that reasoning from scratch on every run, which is why the same résumé scores 7 and then 5. No prompt technique I tested interrupts that, and the most popular one made the yes-or-no answer less repeatable while leaving the number alone.

Then the control took most of the rest. Adding a blank line to my résumé moves the score by 0.262 points. Changing my university moves it by 0.230. Whatever I measured in May, most of it was not about who the candidate is.

What survives is narrow and I think it is solid. First names and career gaps move the score more than a meaningless edit does, in that order, and they are the two signals a hiring tool has the least business reacting to. The rest sit close to the placebo floor of 0.328, and the two weakest, my country and my university, fall under it.

u/kamilc86 got there first, in two sentences, three months before I had the data to agree with him.

There is one thing left to try. If the problem is that the model is making a judgement, the obvious answer is to stop asking it for one. Use it to read the résumé, and score the result in code that a human wrote. That is the next article, and it is the only thing in this whole project that worked.

Sources

The study

• Hiring Bias web app, every evaluation in the project
• Methodology, including how the score changes are measured and audited
• Study repository and raw runs

The three experiments

• Reasoning transplant, all 320 cells with both write-ups and all five scores per arm
• Prompt lab, six strategies compared cell by cell
• Placebo control, the seven meaningless edits and the clean-call Claude re-run
• Paired gaps, the widest single swings, and résumé diffs down to the character

The thread

• The original r/artificial thread, where all three objections were raised

Read the rest of the series: The Blind Spot in the Machine: What 25,500 LLM Evaluations Reveal About AI Hiring Bias and The Frontier Isn't Special: Claude Fable 5 and Hiring Bias.

Explore the open data and reproduce every figure at the Hiring Bias web app and the study repository. The transplant and prompt-lab figures cover the ten models common to those experiments. Prompt-lab stability figures are computed on the 47 of 80 baseline cells with non-zero variance, since 33 cells score identically on every run and cannot be improved upon. All figures are recomputed against the current dataset and differ slightly from the 29 May publication.