Human in the loop, but which loop? Where oversight of AI actually helps in science

The best evidence on this question does not come from biology. The controlled experiment on research ideas was run in natural language processing. The study of analytical variability was run on neuroimaging. The alert-fatigue data comes from hospital prescribing. Nobody has run the equivalent experiment in computational drug discovery.

These fields share the structure that matters: expert judgement applied to a technical artefact, at a point chosen by convention rather than by evidence. The findings transfer as hypotheses, not as proof, and the missing experiment is one worth running on our own field.

The headline result is uncomfortable. Vaccaro, Almaatouq and Malone (2024) ran a preregistered systematic review and meta-analysis of 106 experimental studies reporting 370 effect sizes, and found that on average human-AI combinations performed significantly worse than the better of humans or AI alone. Losses concentrated in decision-making tasks; gains appeared in content-creation tasks.

The moderator analysis is what makes this actionable rather than merely dispiriting. When humans outperformed AI alone, the combination gained. When AI outperformed humans alone, the combination lost. Adding a person to a task they are worse at does not average the two. It drags the result down.

That converts a governance instinct into an empirical question. Before adding a checkpoint, the question is not “is this step important?” but “at this step, is the human better than the system at this judgement?” Those are different questions and they frequently have different answers.

The cleanest demonstration comes from a paired set of studies on scientific work itself.

Si, Yang and Hashimoto (2024) recruited over 100 NLP researchers to write research ideas and blind-review both human and LLM-generated ones. The LLM ideas were judged significantly more novel than the expert ideas, though slightly weaker on feasibility. Read at face value, that is a review checkpoint working: experts assessed the proposals and reached a verdict.

Then the same group executed them. Si, Hashimoto and Yang (2025)recruited 43 expert researchers, randomly assigned them ideas from either source, and had each spend over 100 hours actually implementing the idea and writing it up. The write-ups were then blind-reviewed again. The LLM ideas’ scores dropped significantly more than the human-written ones on all four metrics, novelty, excitement, effectiveness and overall, closing the ideation-stage gap. On many metrics the ranking flipped outright: human ideas now scored higher.

Expand

The reviewers were not weak. They were reviewing at a point where the deciding information did not yet exist.

This is the thesis in one experiment. The reviewers were qualified, motivated and blinded. What they lacked was evidence: at the ideation stage, empirical performance, baselines, ablations and resource requirements were speculative or invisible. A checkpoint placed there cannot discriminate on the criteria that ultimately decide the question, so it discriminates on what it can see, which is fluency and apparent originality. Language models are extremely good at those.

The generalisation is uncomfortable for a lot of deployed review processes. Any gate that asks a human to approve a plan before the plan has produced evidence is measuring the plan’s plausibility, not its correctness.

A second objection to “have an expert sign off” is that expert sign-off is far less determinate than it sounds.

In the Neuroimaging Analysis Replication and Prediction Study,Botvinik-Nezer and colleagues (2020) gave 70 independent teams the same functional MRI dataset and the same nine pre-specified hypotheses. No two teams chose identical analysis workflows. That flexibility produced sizeable variation in which hypotheses came out significant, and it did so even between teams whose intermediate statistical maps were highly correlated. Two analysts can agree substantially about the data and still disagree about the answer.

The constructive half of that result matters just as much: meta-analytic approaches that aggregated across teams did yield significant consensus. Aggregation across analyses outperformed any single expert’s judgement.

For checkpoint design that argues for something other than more approval gates. If one expert’s analytical choices are a draw from a distribution that wide, the useful intervention is to run several defensible analyses and look at the spread, which is the multiverse approach, rather than to have one person initial the one path that happened to be taken. We made a related argument about aggregating agent runs in when agents disagree.

Clinical decision support is the best-documented case of an oversight mechanism collapsing under its own volume, and the numbers are not marginal.

A 2024 systematic review and meta-analysis by Felisberto and colleagues found a pooled drug-drug interaction alert override rate of 90% (95% CI 85.6 to 95.0), drawn from 11 studies covering 570,776 prescriptions within a review of 16 articles. Asingle-hospital evaluation of a DDI system found 88.2% of 38,409 very severe interaction alerts overridden, with false positives traced to an overly broad screening interval and to the system not incorporating patient-specific characteristics.

The detail that makes this more than a cautionary tale: in that same evaluation, prescribers surveyed still found the system useful. The checkpoint was not resented. It was simply generating alerts faster than anyone could meaningfully evaluate them, and the rational response to a channel that is mostly false positives is to stop reading it.

The obvious remedy, better training, does not work. Parasuraman and Manzey (2010), reviewing the empirical literature on automation bias and complacency, reported that both appear in naive and expert participants alike, that complacency cannot be overcome with simple practice, and that automation bias cannot be prevented by training or instructions, producing both omission and commission errors when the aid is imperfect. This is a property of attention under multiple-task load, not a failure of diligence you can exhort people out of.

There is a sharper version of this critique, and it is worth confronting because it lands on the exact requirement most organisations are now trying to satisfy.

Green (2022) surveyed 41 policies prescribing human oversight of government algorithms and argued they suffer two flaws. First, the evidence suggests people are unable to perform the oversight functions the policies ask of them. Second, and consequently, the requirement legitimises deployment of faulty and controversial systems without addressing what is wrong with them, providing a false sense of security and letting vendors and agencies avoid accountability. His proposed remedy is a shift from human oversight to institutional oversight as the central regulatory mechanism.

This sits awkwardly beside regulation. The EU AI Act requires human oversight for high-risk systems under Article 14, and the NIST AI Risk Management Framework expects oversight to be demonstrable. Buyers therefore have to satisfy a requirement that a serious literature says is frequently performative.

The way out is not to argue against the requirement. It is to satisfy it with checkpoints that would survive an audit of whether they actually change decisions. A log showing that 90% of prompts were approved in under three seconds satisfies the letter of oversight and nothing else.

Expand

Rules 1 and 4 follow the cited work. Rules 2 and 3 are ours, argued from the evidence rather than drawn from a single study.

Irreversibility, not uncertainty

The common design gates on model confidence: flag anything the system is unsure about. That produces exactly the DDI failure mode, because uncertainty is abundant and mostly cheap to revisit. Gate instead on what cannot be undone. A speculative intermediate result costs nothing to revisit; committing reagents, animal cohorts, or a trial arm does not. Those are different axes and only one of them tracks cost.

Boundaries where errors go silent

The other systematic error is checking only the final output. By then a wrong identifier mapping or an inappropriate threshold is invisible: it has been laundered into a conclusion that reads perfectly well. Errors have to be caught where they enter, which means checkpoints at chain boundaries, where one stage’s output becomes the next stage’s input. That is the argument we made at length about identifier resolution, and it generalises.

Set the box, do not approve every move

The self-driving laboratory field has already worked out useful vocabulary here, borrowing autonomy levels from autonomous vehicles. Levels run from 0, where humans do everything, to 5, full autonomy. At levels 2 and 3 the system runs design-build-test-learn cycles within constraints humans set, interpreting routine analyses and flagging anomalies for review. A 2024 review in Chemical Reviews reports that levels 2 and 3 account for the vast majority of self-driving laboratories demonstrated to date, and that a true level 5 remains unattained.

The useful part is the shape, not the label. Humans define the space the system may operate in and review what falls outside it. That is a checkpoint on the constraints, which a person can actually evaluate, rather than a checkpoint on every action, which they cannot.

Two design consequences follow, and they are the ones we act on.

The first is that validation should be a separate agent from the one doing the work, running continuously rather than as a terminal gate. Hydra runs a validator whose job is to check that the steps taken match the plan, that results are supported by what was actually retrieved, and that the analysis has not drifted from the question asked. That places checks at the chain boundaries rather than only on the output, which is rule three.

The second is that the human checkpoint we ask for is on the plan and the constraints, not on every step. A scientist sets the direction, the acceptance criteria and the limits; the system executes within them and surfaces what falls outside. That is rule four, and it is why the platform is built around planning and validation rather than a chat that asks for approval at each turn.

Both of those depend on provenance. A checkpoint is only real if the person can see what the system actually did: which databases and versions, which resolved queries, which records were excluded and why. Without that the reviewer is being asked to approve a summary, which is the ideation-stage problem wearing a different hat. Our work on predicting clinical outcome from in vivo data took the strict version of this: endpoints that cannot be resolved into a well-defined schema are excluded and flagged rather than coerced, which costs recall and buys a reviewable result.

Everything above transfers by analogy. The ideation-execution study was NLP. NARPS was neuroimaging. The override data is hospital prescribing. The direct experiment, taking a computational drug discovery workflow, varying where the human checkpoint sits, and measuring downstream decision quality, has not been published as far as we can find.

That is worth stating rather than papering over, and it is a reasonable thing for this field to go and do. Until then the honest position is that checkpoint placement should be treated as a design hypothesis with evidence behind it, not as a governance checkbox, and that the burden of proof sits with whoever wants to add a gate rather than with whoever questions one.

The uncomfortable summary: a checkpoint nobody reads is worse than no checkpoint, because it also supplies assurance. If your approval rate is near 100%, you do not have oversight. You have a log.

• When agents disagree: contradictory outputs across runs
• How to validate agentic AI in bioinformatics
• Will AI replace bioinformaticians?
• Identifier hell: why ontology mapping is the real bottleneck