When agents disagree: handling contradictory outputs across runs of the same question

Start with the mechanism, because it explains why this problem exists at all. A language model does not look up an answer. At each step it produces a probability distribution over possible next tokens and samples from it. Run the same prompt twice and you are drawing twice from a distribution, so a degree of variation is the expected behaviour of a correctly functioning system. Asking why an agent gave two different answers is a bit like asking why two people described the same photograph in different words.

If the differences stopped at wording, none of this would matter. You would paraphrase and move on. What makes run-to-run variation a scientific problem rather than a stylistic one is that the same mechanism can change the logical content of an answer, not just its surface. A different sampled token early in a chain of reasoning sends the rest of the chain somewhere else: a different intermediate assumption, a different threshold, a different conclusion. Two runs can agree on every fact and disagree on what follows from them.

So the useful question is never “why is this nondeterministic.” It is “is this variation cosmetic or load-bearing, and if it is load-bearing, what produced it.”

The instinct when an agent contradicts itself is to make the contradiction go away: raise the sample count, take the modal answer, ship it. That instinct is right for exactly one of the four causes below and actively harmful for another.

Variance across runs is one of the few free signals you get about a pipeline you cannot otherwise inspect. It tells you something about the retrieval layer, the sampling, or the evidence itself. Averaging it away before reading it throws out the diagnostic and keeps the symptom.

Expand

Cause 1 is upstream of the rest. The same voting scheme that rescues cause 3 destroys the information in cause 4. Sources listed in full below.

Cause 1: the model never saw it

This one is upstream of everything else, and it is easy to skip past because it is not really about disagreement at all. A model is trained on a fraction of what exists. Not a random fraction, and not a documented one. Whatever was not in that fraction, the model has no representation of, and when asked it will not report an absence. It will produce the most probable-looking continuation, which is what a hallucination is.

The signature is a run-to-run pattern that looks reassuring and is not: the same confident answer every time, wrong in the same way. Consistency here measures nothing except the stability of a gap in the weights. Sampling more does not help, because you are sampling more thoroughly from a distribution that never contained the answer.

This is the reason retrieval layers exist. If the model cannot be relied on to have the evidence, give it the evidence: retrieve from a real source at query time rather than trusting recall. Which is the right fix, and also the move that introduces the next two causes. Everything below is a consequence of solving this one.

Cause 2: the runs saw different data

This is the most common cause in scientific work and the one most often misdiagnosed as a model problem. If two runs retrieved different record sets, they answered different questions, and no amount of voting over their conclusions will help.

Anthropic’s VirBench work has the cleanest public example. Given the same Ebolavirus sequence query three times, one model returned 106, then 15, then 5 sequences, against an expected 266 (Luebbert, 2026). Vote on downstream conclusions from those three runs and you have averaged three different datasets. The fix is not in the model layer at all: it is deterministic retrieval, version pinning, and a tool layer that echoes the query it actually ran. We covered that in what a good bio MCP server looks like.

Worth knowing that this cause reaches below the model too. Setting temperature to zero removes sampling variance but does not guarantee identical output, because inference kernels are not batch-invariant: dynamic batching changes the order of floating-point reductions, so what else happened to be in the batch can change your result. Thinking Machines Lab demonstrated bit-identical output across 1,000 repeated runs only after replacing the relevant kernels with batch-invariant versions, at a real throughput cost. If you need reproducibility, you have to engineer it.

Cause 3: same data, different path

This is the cause voting was invented for. Given identical inputs, a model can reach the right answer by several routes and the wrong answer by many more. Errors scatter; correct reasoning converges. Sampling several independent paths and taking the most consistent answer, rather than the first one, is the self-consistency method introduced by Wang and colleagues (2023), and it remains one of the highest-value-per-unit-effort techniques available.

The signature is a distribution with a clear mode and scattered outliers, from runs whose inputs you have verified were identical. That verification is the part people skip.

Cause 4: the evidence does not decide

Sometimes the agents disagree because the question is genuinely open. Two defensible readings of the same evidence, two different answers, both correctly reasoned. This is not a failure of the system. It is the system reporting something true about the state of knowledge.

The signature is that disagreement is stable across independent runs and the reasons differ substantively rather than randomly. Run it ten more times and you get the same split, not convergence. Here majority voting is actively harmful: it takes a real 60/40 epistemic split and reports it as a single confident answer, discarding exactly the information a scientist needed. The right output is the split, the reasoning on both sides, and an escalation.

Once teams accept that consensus helps, the reflex is to add samples. Five becomes twenty. The returns fall off a cliff much earlier than expected, for a reason worth stating plainly: repeated samples from one model with one prompt are not independent opinions. They share a prior, a training distribution and a framing. Twenty correlated votes measure that shared prior with more precision. They do not measure the truth.

Independence has to be engineered. In practice that means varying something that actually matters: the evidence each agent sees, the lens each is asked to apply, or the model itself. A panel where one agent checks correctness, one checks whether the result reproduces, and one checks whether the statistics support the claim will catch failure modes that three identical correctness checks never will.

Multi-agent debate is the more structured version of the same idea. Du and colleagues (2024) had multiple model instances propose answers and then critique each other over several rounds, converging on a common answer, and reported improved factuality and reasoning across a range of tasks. The mechanism is not the voting; it is that each agent has to defend its reasoning against a specific objection.

Using one model to grade another’s output has become standard, and the evidence for it is better than sceptics assume. Zheng and colleagues (2023), introducing MT-Bench and Chatbot Arena, found that strong model judges match both controlled and crowdsourced human preferences at over 80% agreement, which is the same level of agreement humans reach with each other. If your bar for a judge is “as good as a second human reviewer,” that bar is cleared.

The same paper documents why you should not stop there. Three biases are well characterised and all three have practical consequences.

The self-enhancement point is the one that bites hardest in practice, because the convenient architecture is exactly the broken one: the same model doing the work and checking it. It will grade its own output generously and give you a validation step that has almost no independent power.

There is a framing change that costs nothing and improves verification more than most architectural work. Do not ask a verifier whether a claim is correct. Ask it to refute the claim, and treat failure to refute as the positive signal.

The reason is asymmetry in how models respond to framing. Asked “is this right?” a model has an easy, agreeable path available and will often take it. Asked “find the flaw in this,” it has to do actual work, and the null result carries information. Pair that with an explicit default: when uncertain, refute. A verifier that has to be convinced is worth several that are willing to agree.

All of these techniques cost tokens, latency, or expert attention. The question is not which is best but which the decision warrants.

Expand

Match the level to the cost of being wrong. Level 1 is the cheapest and the most frequently skipped.

The single highest-return change most teams could make is adopting level 1. Running the same question five times and comparing both the answers and the inputs costs almost nothing and tells you which of the four causes you are in. Without it every subsequent choice is a guess.

Two places in our own work, and they correspond to different rungs.

Consensus in clinical outcome prediction

In our in-vivo augmentation pipeline, language models score compound similarity by target and mechanism and extract endpoint measurements from trial records and literature. Both of those are judgement calls where a single pass is not trustworthy, so they run as a consensus rather than a single verdict. The case study has the method and the numbers: on a retrospective set of 37 drugs across 10 diseases it correctly flagged 84.21% of failures, at 33.33% sensitivity.

That sensitivity number is the honest part and it is directly relevant here. An endpoint that cannot be resolved into the schema is excluded rather than coerced into the nearest match. That is cause 4 handled deliberately: where the evidence does not support a comparison, the pipeline declines to make one instead of voting itself into a confident answer. It costs recall. We think that is the correct trade when a false positive costs a Phase 2.

A validator agent inside Hydra

Hydra runs a separate validator agent whose job is to keep an analysis honest: checking that the steps taken match the plan, that results are supported by what was actually retrieved, and that the analysis has not drifted from the question asked. It is deliberately not the same agent that produced the work, for the self-enhancement reason above.

This is the architectural version of the argument. Planning, execution and validation are different jobs with different incentives, and collapsing them into one agent removes the only check that has any independence. We wrote more about the failure taxonomy this defends against in how to validate agentic AI in bioinformatics.

1. 1Run it five times before you trust it once. Compare the retrieved inputs, not only the final answers. This one step tells you which of the four causes you are in and costs almost nothing.
2. 2Log the inputs, not just the outputs. Record counts, resolved queries and database versions per run. Without them you cannot distinguish a retrieval difference from a sampling difference after the fact.
3. 3Make your judge a different model. Self-enhancement bias means a model grading its own output is close to no check at all.
4. 4Ask verifiers to refute, and default to refuted. Failure to refute is a stronger signal than agreement, and it removes the agreeable path.
5. 5Engineer diversity rather than adding samples. Different lenses, different evidence, different model families. Correlated votes measure your prior, not the truth.
6. 6Report splits you cannot resolve. A stable disagreement with substantive reasons on both sides is a finding. Publishing it as a single answer is a fabrication.

The underlying discipline is simple to state and unpopular to practise. An agent that returns the same answer every time is not necessarily right, and an agent that returns different answers is not necessarily broken. What separates a result you can act on from one you cannot is whether you know which of those you are looking at.

• How to validate agentic AI in bioinformatics
• MCP for scientific data: what a good bio server looks like
• Identifier hell: why ontology mapping is the real bottleneck
• Will AI replace bioinformaticians?