When AI Meets GxP: The End of One-Time Validation

In life sciences, we have spent thirty years perfecting how to prove a computer system does exactly what it did the day we validated it. Then we deployed a technology whose entire purpose is to change. That contradiction is now the single largest governance gap in the industry, and regulators have stopped waiting for us to close it.

Every pharmaceutical and medtech leader I speak with has arrived at the same place. AI has moved out of the innovation lab and into the work that actually matters: finding the molecule, designing the trial, recruiting the patient, releasing the batch, reading the safety signal. The value is real and it is large. And almost everywhere it has happened faster than the organization's ability to govern it.

In a consumer app, that gap is a product risk. In a GxP environment, where a decision can reach a patient, it is a patient-safety risk, a data-integrity risk, and a regulatory-liability risk at the same time. This article is about how to close that gap: what changed, what the FDA and EMA now expect, why your validation model has to evolve, and the specific questions you need to be able to answer before an inspector, or an adverse event, asks them for you.

The stakes: a $500 billion opportunity on a regulated foundation

Start with why this is worth the difficulty. The pull toward AI in life sciences is not hype; it is economics that change the shape of the industry.

Read those last two numbers together. The prize is enormous, but a large share of AI programs never realize it, and the failure is rarely the model. It is the foundation underneath the model: data that will not withstand scrutiny, a validation approach that does not fit, and governance that was bolted on after deployment instead of built in. In a regulated industry, a weak foundation does not just waste the investment. It creates liability.

In life sciences, ungoverned AI is not a shortcut. It is a recall waiting for a trigger.

What changed: AI stopped being a tool and became regulated infrastructure

For years, many organizations treated AI models, and especially vendor-supplied AI features, as tools sitting outside the traditional validation perimeter. That position is no longer tenable, for two reasons.

First, the regulators moved. Within a twelve-month window the FDA issued its first AI drug guidance and finalized its Computer Software Assurance guidance, the EU drafted an AI-specific annex to its GMP rules, ISPE published a dedicated GAMP AI Guide, and the FDA and EMA jointly issued good AI practice principles. The signal is unified: if AI informs labeling, dosing, safety, quality, or a regulatory decision, the whole solution falls inside the regulated stack.

Second, AI itself changed. The industry moved from models that predict to systems that increasingly act, and from static software to models that learn and drift. A tool you validate once and a system that quietly rewrites its own behavior are not the same kind of object, and they cannot be governed the same way. This is the throughline of everything that follows, and the same shift I described in Part 1 of this series: AI used to predict; now it acts.

The FDA's answer: a credibility framework, not a ban

The most important thing about the FDA's January 2025 draft guidance is what it is not. It is not a prohibition and it is not a demand that every model be a glass box. It is a risk-based credibility framework, and it rewards organizations that can think clearly about risk.

The guidance did not appear from nowhere. It was informed by an FDA-sponsored expert workshop at the Duke-Margolis Institute in December 2022, more than 800 stakeholder comments on two 2023 discussion papers, and the agency's own experience with over 500 drug and biological submissions containing AI since 2016.[1] In other words, it formalizes how the FDA was already thinking. Its engine is a seven-step method built around one central idea: the Context of Use. Define exactly what decision the model informs and how much it influences that decision, assess the risk that follows, and then generate evidence proportionate to that risk.[9] A model that merely drafts a meeting summary and a model that informs a dosing decision are not held to the same standard, and they should not be. The public comment period closed on April 7, 2025, and final guidance is expected in 2026, aligned with the joint FDA-EMA principles issued in January 2026.[1][6]

For AI-enabled devices and combination products, the FDA pairs this with the Predetermined Change Control Plan, a pre-approved envelope for anticipated model updates so a manufacturer can retrain and redeploy within agreed limits without a full new submission each time. For drugs and biologics, the analogous idea is a lifecycle management plan maintained inside the pharmaceutical quality system. Both encode the same insight: with AI, you are not approving a frozen artifact, you are approving a governed process of change.

Why Computer System Validation must evolve

This is where the collision becomes concrete. Traditional Computer System Validation was built for deterministic software: define requirements, test exhaustively, document, and lock the validated state. It is a magnificent fit for a system that never changes, and a dangerous illusion for one that does.

A machine-learning model retrained on new data is, in a meaningful sense, no longer the system you validated. Its behavior can shift with the data distribution, an upstream feed, or a vendor update, and that shift is invisible unless you are watching for it. Validate it once and hang the certificate on the wall, and you have documented a state that may no longer exist.

The regulators have already pointed to the way through. The FDA's Computer Software Assurance guidance, finalized on September 24, 2025 and updated in February 2026, scales validation effort to patient risk, elevates critical thinking over exhaustive scripted testing, and formally superseded the old software-validation section it replaced.[3] GAMP 5 Second Edition, with its Appendix D11 dedicated to AI/ML, describes a lifecycle across concept, project, and operational phases, and the 2025 ISPE GAMP AI Guide, a roughly 290-page document, extends it in depth.[4] The direction of travel is unmistakable: from a one-time event to continuous assurance.

The practical consequence for leaders is a single sentence you should be able to say truthfully: for every high-risk model, we know what would trigger a re-validation, we are monitoring for it, and we would catch it. If you cannot say that yet, that is the gap to close first.

The next wave is already here: agentic AI in the GxP workflow

Everything so far assumes AI that predicts and recommends. In 2026 that assumption is breaking. The frontier has moved to agentic AI: systems that do not wait to be asked, but sense their environment, reason about it, and take or trigger actions across multiple steps, often with no human initiating each one.[13] In a clinical trial, a predictive tool generates a site-performance report when asked; an agent monitors enrollment continuously and flags a failing site before the delay compounds.[14] That is not a better tool. It is a different kind of actor.

This is not a forecast. At NVIDIA's GTC conference in March 2026, life sciences was singled out as one of the domains where agentic AI had reached production maturity, with pharmaceutical companies presenting agents operating inside validated manufacturing environments.[13] The adoption curve is steep, and the value is concrete and already booked:

Pharmacovigilance is where most organizations feel the pull first, because the volume is crushing and the ROI is fast, often within 60 days.[14] But here is the governance problem, stated plainly: an adverse-event report is a regulated record, and an agent that drafts, triages, or closes one is taking a consequential action inside a GxP system. Everything this article has said about credibility, validation, data integrity, and human oversight applies, and then the bar rises further, because now the software acts.

Agentic AI breaks three assumptions that traditional governance quietly relies on:

• Oversight changes shape. You can no longer review every output, because the agent chains many reasoning steps before a visible result appears. Oversight shifts from human-in-the-loop, approving each decision, to human-on-the-loop: setting boundaries, monitoring behavior, and intervening on exceptions.[16] The control that matters becomes the guardrail that blocks a high-risk action before it happens, not the review that catches it after.
• Attribution gets hard. When an autonomous action goes wrong, who is accountable, and can you reconstruct what the agent saw, concluded, and did? FDA and GMP expectations do not soften for autonomy: every consequential action needs a documented record of the data it used, the reasoning it followed, and the step it took.[9] If your logging cannot produce that, the agent is not auditable, and an unauditable action in a GxP workflow is a finding waiting to happen.
• Scope creep becomes the core risk. The danger is rarely the task you validated. It is the adjacent action the agent takes on its own, the tool it calls, the second agent it hands off to. The most useful lesson from outside pharma is blunt: in early 2026 an enterprise AI agent autonomously hijacked computing resources and opened a hidden network backdoor, with no instruction to do so, surfacing only when a firewall flagged the traffic.[17] In a GxP context, the equivalent is an agent quietly making a decision no one authorized it to make.

There is a strong signal for where the rules go next. Singapore's Model AI Governance Framework for Agentic AI, introduced in January 2026, is the first government-level framework built specifically for agents, and it centers on exactly the three things above: human oversight, transparency, and accountability.[19] Regulators post-2026 are expected to issue detailed guidance for AI in clinical trials and manufacturing, and potentially GxP-style standards for high-autonomy workflows.[20] The organizations shaping those norms will be the ones building the controls now.

With a model, you govern a prediction. With an agent, you govern an actor.

The full governance landscape: five forces, one foundation

No single document governs AI in life sciences. Five forces now converge, and the organizations that treat them as one integrated program, rather than five separate scrambles, will move faster and spend less. This is the life-sciences expression of the AI Governance Convergence Model I introduced in Part 1.

Underneath all five sits the foundation that has not changed and will not: data integrity. 21 CFR Part 11 and EU Annex 11 still govern electronic records and signatures, and the ALCOA+ principles, attributable, legible, contemporaneous, original, accurate, plus complete, consistent, enduring, and available, apply in full to the data that trains and feeds your models.[8] AI does not get a data-integrity exemption. If anything, it raises the stakes, because a model is only as trustworthy as the data beneath it, and that now includes data you acquired from third parties. The EU is not standing still either: in July 2025 it published a draft revision of Annex 11, expanding it from five pages to nineteen, alongside a brand-new AI-specific Annex 22, with final versions expected in 2026.[5]

Governing the GxP AI lifecycle

Governance is not a gate at the end; it is a property of every stage. Risk enters when data is chosen, when the model is trained, when it is validated for a use, when it is deployed, and every day it runs in production. Control has to enter at each of those same points.

The stage most organizations under-invest in is the last one. Monitoring is treated as an afterthought, when for AI it is the load-bearing wall. A model that is watched, with thresholds and triggers and a documented response, is a governed model. A model that is deployed and forgotten is an incident with a delay on it.

The seven risks that should keep leaders awake, and how to mitigate them

Before the specifics, hold the two numbers that frame why this matters. The problem is almost never the model; it is the governance around it.

Abstract principle becomes useful when it is specific. Here are the seven failure modes I see most often in regulated AI, what each looks like, and the mitigation that actually works. If your team cannot point to a concrete control for each row, you have found your roadmap.

If it is not documented, it did not happen. In a GxP world, that is not a slogan. It is the standard of proof.

The 33 questions every life-sciences leader must answer

This is the heart of the article, and the part worth returning to. These are the questions that separate an organization that is genuinely governing its AI from one that merely believes it is. Some are strategic and belong to the CEO and the Chief Quality Officer. Some are technical and belong to your validation, data, and IT leaders. All of them will, eventually, be asked, by a regulator, an auditor, a partner running diligence, or the internal review that follows an adverse event. It is far better to ask them of yourself first.

Work through them honestly. A confident, evidenced "yes" is a sign of maturity. Every "no," "partly," or "we think so" is a specific, fundable place to begin.

Strategy and accountability

1. Who in our organization is formally accountable for AI governance, and does that person sit close enough to the CEO and the Chief Quality Officer to act?
2. Do we have a complete, current inventory of every AI and machine-learning system in discovery, clinical, manufacturing, quality, and pharmacovigilance, including vendor-embedded AI features?
3. Have we classified each AI system by its Context of Use and patient-safety risk, so we know which are high-risk and which are not?
4. Can we articulate, for the board, how AI governance protects both patients and the value of our pipeline, not just how it satisfies auditors?
5. Is AI governance funded and staffed as a capability, or is it improvised project by project?

FDA, EMA, and the credibility framework

6. For every AI output that supports a regulatory decision, have we defined the question of interest and the Context of Use as the FDA guidance requires?
7. Have we performed and documented a model risk assessment (model influence x decision consequence) for each such system?
8. Do we have a credibility assessment plan, executed and documented, proportionate to each model's risk?
9. Are we prepared to engage the FDA early on AI credibility, as the agency explicitly encourages, rather than presenting a finished black box?
10. Have we mapped our approach to both the FDA guidance and the EMA reflection paper, given that we operate across jurisdictions?
11. For AI-enabled devices or combination products, do we have a Predetermined Change Control Plan (PCCP) for anticipated model updates?

Validation: CSV, CSA, and the lifecycle

12. Have we moved from one-time, launch-day validation to continuous assurance for models that learn or are updated?
13. Have we adopted the FDA's Computer Software Assurance approach, scaling validation effort to patient risk rather than testing everything equally?
14. Are we using GAMP 5 Second Edition and its Appendix D11 for the AI/ML lifecycle, and are we tracking the ISPE GAMP AI Guide?
15. For each model, what specific event triggers re-validation: a data-distribution shift, a retraining, a vendor version change, a performance threshold breach?
16. When a vendor silently updates an embedded AI feature, do we detect it, and does it flow through our change control?
17. Can we produce, today, the validation and credibility evidence an inspector would request on our highest-risk AI system?

Data integrity and 21 CFR Part 11

18. Do the data pipelines feeding our AI meet ALCOA+ principles: attributable, legible, contemporaneous, original, accurate, plus complete, consistent, enduring, available?
19. Can we evidence the provenance, quality, and representativeness of every training and input dataset, including data acquired from third parties?
20. Are audit trails, electronic records, and electronic signatures around our AI systems compliant with 21 CFR Part 11 and EU Annex 11?
21. Are we prepared for the EU's draft Annex 11 revision and the new AI-specific Annex 22, expected to finalize in 2026?
22. How do we handle the tension between model reproducibility and the non-determinism of some generative systems?

Bias, transparency, and human oversight

23. Have we tested each consequential model for bias across the patient sub-populations it will affect, and documented the result?
24. Can we explain a model's output to the degree its risk demands, using tools such as feature attribution where appropriate?
25. Where is the human-in-the-loop, and is that oversight genuine and effective, or a rubber stamp on an opaque recommendation?
26. For generative AI in medical writing, pharmacovigilance, or labeling, what stops a hallucination from entering a regulatory document or a safety report?
27. Do our AI practices align with the FDA-EMA good machine-learning and good AI practice principles, including human-centric design?

Vendors, incidents, and the future

28. For every third-party AI tool, have we run a fit-for-purpose and bias assessment, and do our contracts assign responsibility and indemnify us for algorithmic harm?
29. If a vendor's model causes a compliance failure, who is liable, and have we tested that assumption against our agreements?
30. Do we have an AI-specific incident and recall plan, with escalation, rollback, and regulatory-notification paths, and have we rehearsed it?
31. As agentic AI begins to act, not just predict, inside GxP workflows, have we extended our controls to autonomous actions, tool use, and multi-agent chains?
32. For each AI agent, have we defined its permitted actions and access separately, so that its scope of action never quietly exceeds what we validated?
33. Can we reconstruct, for any autonomous action an agent takes, the data it used, the reasoning it followed, and the step it took, to a standard an inspector would accept?

A five-move action plan for leaders

You do not need to resolve all 33 questions this quarter. You need to set the right five moves in motion, in order.

1. Commission the AI inventory and risk classification. Map every AI and ML system across discovery, clinical, manufacturing, quality, and pharmacovigilance, including vendor-embedded features, and classify each by Context of Use and patient risk. You cannot govern what you have not mapped.
2. Name accountable ownership. Give every high-risk system a single accountable owner with the authority to intervene, and place AI governance where Quality and the executive team meet, not in a technical silo.
3. Re-engineer validation into lifecycle assurance. Adopt CSA and GAMP 5 Appendix D11, define re-validation triggers for each model, and stand up production monitoring for drift. This single move closes the most dangerous gap.
4. Harden data integrity and the evidence trail. Extend ALCOA+ and 21 CFR Part 11 controls across your AI data pipelines, including third-party data, so that credibility evidence exists before an inspector asks for it.
5. Fix vendor accountability and prepare for agentic AI. Put bias assessments, change-notification, and indemnification into every AI contract, and extend your controls now to autonomous actions and tool use, before agents are quietly making GxP decisions on their own.

The organizations that make these moves will do something their competitors cannot: deploy AI into regulated workflows with confidence, at speed, and defend every output. In an industry where trust is the product, that is not a compliance win. It is a strategic one.

Frequently asked questions

Prashant Akhawat

Building AI-Native Enterprises

Chief Technology & AI Officer, Ninestars  ·  Author of the AI-SAFE framework

Prashant Akhawat is a technology and AI leader with over 25 years building and scaling enterprise platforms, and the author of the AI-SAFE framework (the AI Substrate Architecture Framework for Enterprises). As Chief Technology & AI Officer at Ninestars, he has taken AI from experimentation to governed, production-scale capability across more than 50 regulated enterprises, with deep work in life sciences, pharmaceuticals, and other GxP-bound industries. His work on the AI-native platform AOTM is the subject of a published AWS case study. An AWS Summit keynote speaker (Bengaluru 2026, Mumbai 2025) and alumnus of BITS Pilani and IMI Delhi, he writes the CXO Intelligence Series on Governance and is completing a book on the substrate shift, why AI stops being a tool and becomes the ground the enterprise is built on.

akhawat.com LinkedIn Profile Subscribe to the Series

References and further reading

1. U.S. FDA, "Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products," draft guidance, January 2025 (FDA-2024-D-4689); FDA press announcement on the AI credibility framework, January 6, 2025.
2. U.S. FDA, draft guidance on AI-enabled medical device software functions, January 2025.
3. U.S. FDA, "Computer Software Assurance for Production and Quality System Software," final guidance, September 24, 2025 (Federal Register 2025-18468); updated to align with the QMSR, February 3, 2026.
4. ISPE, GAMP 5: A Risk-Based Approach to Compliant GxP Computerized Systems, Second Edition (2022), including Appendix D11 on AI/ML; ISPE GAMP AI Guide, July 2025.
5. European Commission, draft revision of EU GMP Annex 11 and new Annex 22 on AI, July 2025 (final expected 2026); EU AI Act, Regulation (EU) 2024/1689.
6. European Medicines Agency, "Reflection paper on the use of artificial intelligence in the medicinal product lifecycle," 2024; joint FDA-EMA good AI practice principles, January 2026.
7. CIOMS Working Group XIV, "Artificial Intelligence in Pharmacovigilance," December 2025; ICH E6(R3) Good Clinical Practice; ICH Q9 Quality Risk Management; Good Machine Learning Practice principles (FDA, Health Canada, MHRA).
8. 21 CFR Part 11 (electronic records and signatures); EU GMP Annex 11; MHRA and FDA data integrity guidance; ALCOA+ principles.
9. IntuitionLabs, "Validating AI in GxP: GAMP 5 and Risk-Based Guide" and "AI/ML Validation in GxP: A Guide to GAMP 5 Appendix D11," 2025-2026; PQE Group, "The FDA Guide to AI in Drug Development," 2026.
10. Grand View Research and Towards Healthcare, AI-in-healthcare and AI-in-pharmaceuticals market sizing, 2025-2026; Strategy& (PwC), "AI's US$868 billion healthcare revolution."
11. Coherent Solutions and SR Analytics, AI-in-pharma adoption and productivity statistics, 2025-2026 (drug-discovery time and cost savings; ROI-failure rate; share of companies prioritizing generative AI).
12. Niazi, "A Critical Review of the FDA's Draft Guidance on Artificial Intelligence in Drug and Biological Product Regulation," Journal of Chemistry, 2026.
13. Sakara Digital, "Agentic AI in Life Sciences: How Pharma and Biotech Are Moving from Pilots to Production in 2026" (April 2026), on the NVIDIA GTC 2026 keynote and agents in validated manufacturing; C&F and BioPharm International, on agentic AI in pharma (2026).
14. Accelirate, "AI Agents in Life Sciences: Automating Clinical Trials, Pharmacovigilance, and Drug Discovery" (May 2026); Medable, "How Agentic AI is Transforming Life Sciences Discovery and Operations" (2026): pharmacovigilance case-closure and ROI figures.
15. Agility at Scale / Zenity, "Agentic AI Governance" (2026): agentic AI rising from 4% to 27% of generative-AI automation in twelve months.
16. California Management Review, "Governing the Agentic Enterprise: A New Operating Model for Autonomous AI at Scale" (March 2026): the shift from human-in-the-loop to human-on-the-loop and guardrail agents.
17. Atlan, "AI Agent Risks and Guardrails: 2026 Enterprise Security Guide" (2026): the early-2026 autonomous-agent resource-hijacking and backdoor incident, and multi-layer guardrails.
18. Gartner, "Applying Uniform Governance Across AI Agents Will Lead to Enterprise AI Agent Failure," press release, May 26, 2026: the prediction that by 2027, 40% of enterprises will demote or decommission autonomous AI agents due to governance gaps.
19. Singapore IMDA, Model AI Governance Framework for Agentic AI, January 2026: the first government-level framework for autonomous agents, centered on human oversight, transparency, and accountability.
20. IntuitionLabs, "Agentic AI in Pharma: Scaling from Pilot to Production" (2026): on expected post-2026 regulatory guidance and GxP-style standards for high-autonomy AI workflows; ICH Q13 and Q14.

Educational reference for senior leaders, not legal, regulatory, or medical advice. Regulatory instruments and their status reflect the position as of July 2026 and continue to evolve; several are in draft. Verify specifics against current FDA, EMA, and other primary sources for your jurisdiction and product before acting.