How do you validate a non-deterministic agent under GxP?

Qualify it as a method or a process rather than as software, whenever the output feeds a regulated decision. Concretely, that means five things.

1. 1Define the context of use narrowly, and enforce it.Not an agent for regulatory writing, but drafts section 3.2.P.5.1 from these three source document types. A method has a range. Give the agent one.
2. 2Set acceptance criteria on the distribution, not the instance.Accuracy and precision across a fixed evaluation set, with thresholds fixed before testing, exactly as you would for an assay. The same answer twice is not the bar, and never was for a bioassay either.
3. 3Pin the model version and put it under change control.This is the control CSV genuinely lacks. A vendor-hosted model that updates silently is an unqualified change to a validated system. Two ways to close it: host the weights yourself, so the artifact is yours and cannot move without your change request; or contract for it, requiring the vendor to serve a named, dated model version through a pinned API endpoint, with notice periods and a deprecation window written into the agreement. Either way the version becomes a controlled item with a number you can put in a deviation report.
4. 4Run continued verification, not one-time qualification.Sample live outputs on a schedule, trend them against your criteria, and set the drift threshold that triggers requalification before you need it.
5. 5Name a qualified human reviewer, and document the qualification.Not a human checks it, but a named, trained, accountable person. This is what the first FDA warning letter on the subject was actually about.

Not every agent needs all five

Applying the full set to every deployment is how validation budgets get spent on things that do not matter. The regulatory direction of travel is the opposite: FDA finalised Computer Software Assurance for Production and Quality System Software on 24 September 2025, updated February 2026, which replaces uniform validation with assurance effort scaled to what the software actually decides. Three questions size it: how severe the potential harm is, how likely the failure is, and whether you would detect it if it happened.

One capability lands in three different places depending on the decision it feeds. An agent summarising literature for an internal team meeting is low risk and needs little more than a record of what it did. The same agent informing which programme to pursue is medium risk and needs a review checkpoint with a named reviewer. The same agent again, drafting text that enters a regulatory submission, is high risk and needs all five controls, because the output now reaches a decision about patients. Nothing changed about the model. The only variable is what happens downstream of its output.

Four things hold at every level, and they are cheap: keep a record of what the agent did, be able to show the data behind it was not altered, be able to reproduce a result when asked, and manage updates deliberately rather than incidentally. Two existing documents give you a vocabulary an auditor already accepts: the NIST AI Risk Management Framework (AI 100-1, January 2023) for the risk language, and the MHRA GxP data integrity guidance (March 2018) for ALCOA+, which your quality unit has been applying to records for years and which applies unchanged to records an agent produced.

The premise deserves testing before it is accepted, so here is the record.

The first FDA warning letter to name an AI agent

It went to Purolea Cosmetics Lab on 2 April 2026, warning letter 320-26-58, and it contains a dedicated section headed Inappropriate Use of Artificial Intelligence in Pharmaceutical Manufacturing. The firm used AI agents to create drug product specifications, procedures, and master production and control records. FDA’s finding: “If you use AI as an aid in document creation, you must review the AI generated documents to ensure they were accurate and actually compliant with CGMP. Your failure to do so is a violation of 21 CFR 211.22(c).” The corrective expectation is explicit: any output or recommendation from an AI agent must be reviewed and cleared by an authorised human representative of the firm’s quality unit.

Read the citation again. 21 CFR 211.22(c) is the quality unit regulation, not a computerised-systems clause. The violation was that nobody reviewed the output. Whether the model was deterministic never entered into it.

There is a detail in that letter worth sitting with. The firm had not performed process validation, and explained that it was unaware of the requirement because the AI agent it used had not told it. That is the whole argument in one sentence: not a machine behaving unpredictably, but a company treating a tool as an accountable party. The firm has since committed to cease production and distribution of drugs at the facility.

Knight Capital, and entirely deterministic software

Unsupervised automation is not a pharmaceutical problem, and the clearest evidence for that sits outside the industry. Finance automated earlier, at higher speed and with tighter feedback loops, so its failures arrived first and were investigated in public. Two of them are worth borrowing, because neither involved a model that behaved unpredictably.

Knight Capital lost more than $460 million in 45 minutes on 1 August 2012. Dormant code from 2005 was reactivated when a technician copied a new deployment to seven of eight servers; the router “rapidly sent more than 4 million orders into the market when attempting to fill just 212 customer orders”. The SEC charged it as a market access rule violation and the firm paid $12 million. Deterministic software throughout. A change-control failure, and a second-reviewer failure.

Zillow Offers, and the verification that never ran

Zillow wound the business down on 2 November 2021 with a $304 million inventory write-down and roughly a quarter of the workforce cut. CEO Rich Barton’s explanation was that “the unpredictability in forecasting home prices far exceeds what we anticipated”. A model whose real-world error was never continuously verified against outcomes.

Three failures, three different technologies, one shared shape: nobody was watching the output against a standard. That is a control failure. It is not a property of probability.

Here is what makes the panic strange. Pharma has validated non-reproducible things for decades. It simply does not file that activity under computer system validation.

Method validation already validates scatter

ICH Q2(R2), adopted 1 November 2023, defines precision as the closeness of agreement, the degree of scatter, between a series of measurements obtained from multiple sampling of the same homogeneous sample under prescribed conditions. Its acceptance test is that the observed interval should be compatible with the corresponding accuracy acceptance criteria unless otherwise justified. Nothing in that guideline demands identical output. It demands that variability sit inside a pre-declared interval, which is precisely the test an agent needs.

Process validation already handles drift

FDA’s process validation guidance of January 2011 defines three stages ending in Continued Process Verification, whose goal is “continual assurance that the process remains in a state of control (the validated state) during commercial manufacture”. Pharma has had a legal answer to “it changes after you validate it” for fifteen years.

Regulators already accept probabilistic evidence

FDA’s guidance on Bayesian statistics in medical device trials, final February 2010, accepts that a hypothesis has been demonstrated with reasonable assurance if its posterior probability is large enough. The adaptive designs guidance, final November 2019, permits the design itself to change mid-trial provided the details are completely specified before the trial begins. Pre-specification, not determinism, is the control.

While GMP and banking were writing generative AI out of scope, a different part of FDA had been building a framework for models that change, and had been at it since 2019. It sits in device regulation, which is why it rarely enters the GMP conversation even though the problem is recognisably the same one.

What SaMD is, and what it is not

Software as a Medical Device is defined by the International Medical Device Regulators Forum as software intended to be used for one or more medical purposes without being part of a hardware medical device. A diagnostic algorithm reading mammograms is SaMD. An agent drafting a CMC section, reconciling a batch record or summarising a deviation is not.

That distinction is worth stating plainly, because the two vocabularies are converging in conference talks and they are not the same regime. FDA’s own paper draws the line explicitly: non-device software functions are outside its scope, including software for administrative support of a health care facility, for maintaining or encouraging a healthy lifestyle, for electronic patient records, for transferring, storing, converting or displaying data, and for certain limited clinical decision support. Borrowing this framework for a manufacturing agent is an analogy, not a compliance route. It is a good analogy, and it is not a submission.

Risk is a property of the decision, not the technology

The framework categorises software on two axes: what the information is used for, and how serious the situation is. Those two questions produce four categories, from I at the lowest risk to IV at the highest. Notice that the model itself appears nowhere in the grid.

This is the same logic as the risk-based sizing above, drawn as a grid. The identical algorithm sits in category IV or category I depending on whether it treats a critical condition or informs a decision about a non-serious one. Any organisation trying to tier its own agents can steal this table directly, substituting its own axes for the clinical ones.

FDA has a word for deterministic, and it is not the requirement

The paper defines a locked algorithm as one that provides the same result each time the same input is applied to it and does not change with use, and then spends the rest of its length on the algorithms that are not locked. It states the consequence without flinching: given the same inputs before and after an update, the output of an adaptive algorithm may differ. A regulator wrote down a category for output that changes between runs, described how it would oversee it, and published that in 2019.

Expand

Pre-specify the envelope, then the method for moving inside it

The mechanism at touchpoint 3 is the part worth stealing. A manufacturer declares two things in advance. The pre-specifications are what the model is expected to become as it learns, drawing what the paper calls a region of potential changes around the original specification. The algorithm change protocol is the step-by-step method for getting there safely: the data, the procedures, and the evidence that the thing is still safe and effective afterwards. Declare the envelope, declare how you move inside it, and changes within that envelope stop being unqualified.

Two caveats keep this honest. The 2019 document says of itself that it is being issued for discussion purposes only and is not a draft guidance, so nothing in it binds anyone. And it took five years to become policy: the mechanism landed as FDA’s Predetermined Change Control Plan guidance in December 2024, which is the same idea with a submission attached. That is the pace at which a regulator turns a good idea into a rule, and it is worth remembering before waiting for one.

Which makes what happened next harder to explain. In the space of ten months, the two regimes closest to the problem for a manufacturer, GMP and model risk management, both responded to generative AI by putting it outside the scope of the framework rather than extending the framework to cover it. The device track had already shown the extension was possible.

Expand

Europe proposed a ban, and is now walking it back

The EU’s draft Annex 22 on Artificial Intelligence went out for consultation on 7 July 2025, closing 7 October 2025. Its scope section is blunt. Deterministic models, which given identical inputs provide identical outputs, are covered. Models with a probabilistic output are not covered and “should not be used in critical GMP applications”. The same sentence is applied to generative AI and large language models.

The draft contains the right instinct elsewhere: that the acceptance criteria of a model should be at least as high as the performance of the process it replaces, and that metrics should be defined for monitoring drift in input data. That benchmark is exactly right, and it is not a reproducibility test. It is a comparative performance test, which is the logic of method validation.

ISPE objected formally on 7 October 2025, arguing that restricting the annex to static models will restrict innovation and asking that advanced models be allowed in critical GMP applications when justified by science and quality risk management. That objection has landed. EMA convened a multistakeholder expert workshop on 30 June and 1 July 2026 to gather evidence on possible control and mitigation measures such as guardrails, and to shape a risk-based approach for manufacturers wanting to use these technologies in GMP applications. The 2025 consultation, EMA notes, suggested support for enabling generative AI and LLMs in medicines manufacturing.

US banking superseded SR 11-7, and carved agentic AI out

The comparison everyone reaches for is banking, and with reason. Supervisory letter SR 11-7, issued 4 April 2011, opens from the premise that “All models have some degree of uncertainty and inaccuracy because they are by definition imperfect representations of reality”. Its answer was three-part: disciplined development, effective challenge, defined as critical analysis by objective, informed parties that can identify model limitations and produce appropriate changes, and ongoing monitoring including outcomes analysis, which compares model outputs to corresponding actual outcomes. Fifteen years of governing models nobody expected to be reproducible.

On 17 April 2026 that guidance was replaced. SR 26-2, issued jointly by the Federal Reserve, OCC and FDIC as OCC Bulletin 2026-13, supersedes SR 11-7 outright. Its third footnote is the part pharma should read:

“Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance. Nonetheless, a banking organization’s risk management and governance practices should guide the determination of appropriate governance and controls for any tools, processes, or systems not covered in this document.”

So the most mature model risk framework in any industry was revised at exactly the moment it became relevant, and the revision leaves out the models everyone is asking about. It still covers traditional statistical models and non-generative AI. A request for information on banks’ use of AI is promised, with no date.

Read the second sentence of that footnote again, though, because it is the useful part. Firms are told to use their own governance practices to set controls for anything the guidance does not cover. That is a delegation, not a moratorium, and Europe is arriving at the same place from the opposite direction. Whichever way either consultation lands, the controls are yours to design.

Two things are true at once, and most of the discourse picks one. The first is that stochasticity is controllable. A model version can be pinned, an evaluation set can be fixed, an acceptable interval can be declared in advance and then watched indefinitely to see whether performance stays inside it. That is not a novel proposal. It is what pharma already does to a bioassay and to a manufacturing process, neither of which gives the same answer twice. Nothing about a probability distribution puts a system beyond control. For what that looks like on one concrete workflow, the certificate of analysis handoff with a contract manufacturer, see can an AI agent take over CDMO paperwork.

The second is that this really is a new category of computation, which is why the legislation is lagging rather than merely slow. The clearest evidence is Europe’s own handling of it: an exclusion written into the Annex 22 draft in July 2025, and within a year a multistakeholder workshop convened to work out the guardrails that would let those models back in. That is not a regulator being cautious. It is a regulator drafting against a moving object, and the same hesitation shows in a US banking revision that declined to extend fifteen years of model risk practice to the models everyone is asking about.

So the honest position is that nobody has the definitive answer yet, this article included. What it offers instead is a transfer: controls that already work on systems that vary, taken from the qualification logic of an analytical method, the continued verification of a process, the version discipline of change control, and the independent challenge of model risk management, and applied to agents until something purpose-built replaces them. Treat it as a defensible estimate of how to control an agentic system in production, not as a standard.

And on the evidence, the thing that has actually caused enforcement, losses and shutdowns is not that the machine was uncertain. It is that nobody was accountable for reading what it said. That failure predates AI by a century, and pharma already knows its name.

Related reading: how to validate agentic AI in bioinformatics covers the technical failure taxonomy underneath these controls, and where to put the human checkpoint covers which review positions actually change outcomes.