AI in the GMP environment: What the Annex 22 conference means for practice

When convincing answers are factually wrong

In June 2026, the UK's Medicines and Healthcare products Regulatory Agency (MHRA) reported AI-assisted inspection responses containing fabricated references and inappropriate references to regulatory requirements. In one case, its time spent on review and feedback increased from around four to more than twenty hours. The texts delayed the handling of significant deficiencies. [8]

The case illustrates a central challenge for AI in the GMP environment: fluent, convincing outputs can be factually wrong and create additional work rather than reduce it. The value of an AI application therefore also depends on how reliably errors are detected and how much review and correction effort remains.

This tension also shaped the ECA Annex 22 / AI Conference held in Copenhagen on 7 and 8 October 2026, preceded by the introductory course on 6 October. Perspectives from inspection, industry and consultancy came together. The EMA workshop report dated 2 October 2026 provides a corresponding account of the regulatory discussion: the focus is on evidence of fitness for intended use and risk-based controls throughout the lifecycle. [7]

This article considers operational applications in the GMP environment. After a brief introduction to the underlying concepts, we outline the regulatory framework and use a hypothetical deviation investigation assistant to work through assessment, evidence generation and operation. Publicly described applications complement this example. The regulatory assessment of AI as a medical device, and of AI models in medical devices, is addressed in a separate article.

The tasks different AI applications can perform

Artificial intelligence (AI) encompasses different methods that enable computer systems, for example, to recognise patterns, make predictions or generate content. Machine learning (ML) is an important subset. Models are trained on data to use patterns and statistical relationships for a particular task. Deep learning is a form of machine learning that uses neural networks with many processing layers. It underpins numerous image-analysis applications and today's large language models. [1, 19] Such networks learn relevant features, for example in images, directly from training data rather than relying on people to specify them. This makes them powerful but makes individual results harder to understand; the draft Annex 22 therefore addresses explainability and confidence in outputs. [6]

A common use of machine learning in the GMP environment is automated visual inspection. Syntegon describes an inspection system used at Amgen which, according to the manufacturer, has been validated and uses deep learning to distinguish air bubbles at the syringe stopper from foreign particles. [18] For such applications, the question is which defects a model detects under which imaging conditions. Missing a critical defect has different consequences from incorrectly rejecting an acceptable container. Error types, rare defects and different operating conditions therefore need to be considered separately.

Assessment in the GMP environment also requires an understanding of whether a model continues to learn during use. During training, internal model values known as parameters are adjusted so that the model performs its task as well as possible. In a static model, the parameters established during training remain fixed in operation. An adaptive or dynamic model can also adjust these parameters during operation on the basis of new data. A model operated statically can still be retrained later and replaced by a tested new version. Machine learning therefore does not automatically mean continuous learning during use. [6]

Repeatability of outputs is a separate issue: even a static model can produce different results from the same input. Deterministic means that identical inputs and system conditions produce identical outputs. [6, 12]

Generative AI produces content. Large language models (LLMs) can summarise texts, answer questions and prepare wording. Autoregressive language models use the input and the text generated so far to calculate probabilities for the next text element, known as a token. [1]

Probabilistic modelling does not necessarily mean random output: a fixed selection rule can produce repeatable responses, whereas sampling is influenced by factors including temperature. Technical variations can occur even at temperature 0. [12] Our takeaway from the conference discussions is that a system with repeatable outputs can still produce factually incorrect results. Repeatability and factual correctness therefore need to be assessed separately.

In knowledge assistants, retrieval-augmented generation (RAG) combines information retrieval with text generation: the system searches a document collection for relevant material and provides it to the language model as context. This allows it to use internal or current information that was not included in model training. RAG is therefore a system architecture, rather than a distinct type of model. [2]

The ability to act also varies. An assistant can display information or prepare recommendations. An AI agent can additionally call tools and perform actions within its permissions. Its assessment must therefore also consider unwanted changes and the permissions and interfaces actually granted. Figure 1 puts these concepts and characteristics into context.

Figure 1: AI methods and application characteristics. Static operation and repeatable outputs need to be assessed separately; RAG connects information retrieval and text generation. Our own schematic illustration based on [1, 2, 6, 12, 19].

What already applies and how Annex 22 is developing

Validation requirements already apply to the use of AI in GMP-regulated activities. The basis is Annex 11 of the EU GMP guidelines on computerised systems: the application is to be validated, with the scope determined by a justified and documented risk assessment. [4] A final Annex 22 is not a prerequisite. Where a system replaces a manual activity, Annex 11 states that product quality, process control and quality assurance must not decrease, and overall risk must not increase. [4]

Our article “AI in GMP and MedTech Companies: Which Regulations Apply and When Is Validation Required?” explains which frameworks apply to different uses and when validation is necessary. It also examines the distinction from medical device legislation and the EU AI Act.

The 2025 draft Annex 22 applies to static ML models with deterministic outputs used in critical GMP applications in the manufacture of medicinal products and active substances. Generative AI and LLMs are still excluded from this scope; the current draft does not provide for their use in critical GMP applications. For non-critical applications of generative AI, qualified users are expected to ensure that outputs are suitable. The consultation ended in October 2025; a final version was still outstanding at the article's editorial cut-off date. [6]

The EMA workshop report records discussion of an extension to adaptive, probabilistic and generative AI supported by documented, robust and risk-based control strategies. It identifies the Drafting Group's review of workshop contributions as the next step in the further revision of Annex 22. [7] Another takeaway from the conference discussions is the need to distinguish the current stage of development from binding requirements. Companies should assess the intended use, capabilities and risks of their specific application in a traceable way.

For a company's own use, this means specifying the task the application is to perform and how its outputs will be used in the GMP process. This provides the basis for deriving possible errors, appropriate controls and testable requirements. We explain this step below using a simplified example.

Assessing an AI application through a deviation investigation assistant

Assessment begins with the intended use. It describes the task an application is to perform, the conditions under which it will be used and how its outputs will feed into the workflow.

We consider a simplified, hypothetical example from equipment cleaning in pharmaceutical manufacturing: a deviation is documented during a cleaning cycle. An AI assistant helps the responsible staff establish the facts. It brings together the cleaning record, relevant equipment logbook entries, the requirements applicable to the event and potentially comparable previous deviations. From these, it prepares a factual summary with source references, presents conflicting information side by side and flags open questions. Previous cases provide material for comparison; their causes and actions must not be transferred to the current event without assessment.

The output remains identifiable as an AI-generated draft and is reviewed by qualified staff against the original documents. Assessment of causes and potential impacts, selection of actions and approval remain the responsibility of the designated personnel. Within its intended functionality, the assistant has read-only access to source data; it neither creates nor changes GMP records and makes no release decision.

Catalent's Qai provides a real-world example of this type of support. According to the published descriptions, the system makes the company's own quality data, including previous deviations and CAPA, accessible and supports the preparation of investigation drafts. Catalent states that every output undergoes documented review by quality experts. [16, 17] The cleaning scenario and associated test cases are our own hypothetical example; the sources do not describe such a cleaning case.

These functions show why a precise task description matters: a previous case involving different equipment may be applied incorrectly, an unresolved cause may be presented as confirmed, or a contradictory logbook entry may be omitted. Table 1 relates the functions to possible errors and suitable test cases.

Table 1 Hypothetical test cases for the deviation investigation assistant

The intended use also helps determine the choice of technology: improved search may be sufficient to find documents; identifiers, mandatory fields and document versions can be checked using rules. The potential value of generative AI here lies in turning dispersed, partly unstructured information into a factual summary supported by evidence. This differs from merely improving the wording of an already reviewed report or recommending a cause and appropriate CAPA. It must be demonstrated that the chosen support is reliable and reduces the overall review effort.

ICH Q9(R1) grounds quality risk management in scientific knowledge and patient protection. Effort, formality and documentation should be proportionate to risk; uncertainty, importance and complexity influence the degree of formality required. [5] The consequences of errors and the likelihood of detecting them therefore need to be assessed for the deviation investigation assistant.

Criticality depends on the potential consequences of errors and the actual use of the outputs. Even a summary can influence a GMP-critical decision, for example if an unsubstantiated cause appears to be confirmed and prematurely narrows the investigation. Human review therefore does not automatically make an application non-critical. Omissions deserve particular attention: an unreported discrepancy cannot be detected solely by checking the references that are displayed; the relevant source material must also be compared. The 2025 draft Annex 22 does not provide for generative AI and LLMs in critical GMP applications. [6] The hypothetical example therefore describes testable functions and boundaries; whether a specific use is acceptable must be assessed in light of the intended workflow, its risks and the demonstrated effectiveness of the controls.

Figure 2 summarises the assessment steps from task definition through to routine operation; the following sections examine them in more detail using the deviation investigation assistant.

Figure 2: Assessment connects task definition, risks, evidence, controls and operation. Experience and changes feed back into reassessment. Our own schematic illustration based on [7].

The data, evidence and controls required

The data the deviation investigation assistant needs for its task

The assistant's data requirements follow from its task. An accurate document or a validated source system alone does not show that all the information needed is available. What matters is content and scope, together with the equipment, products, cleaning types and deviations for which the assistant is to be used. The NIST AI Risk Management Framework also addresses the relationship between data, application context and intended task. [20] Likewise, a validated interface to a manufacturing execution system (MES) demonstrates correct data transfer only within the tested scope. Whether the transferred information is sufficient for the AI assistant's particular task requires a separate assessment.

Which information is needed? To prepare the factual summary, the assistant needs the deviation notification, the associated cleaning record, relevant equipment logbook entries and the requirements applicable to the event. Previous deviations and associated CAPA supplement the search where they may be relevant to the question. Equipment and event identifiers, chronology, document versions and investigation status must be assigned unambiguously. The permitted scope of data, access rights and transfer routes must be defined before operational data are used. [10]

The information must be prepared in a way that distinguishes observed facts, suspected causes and confirmed investigation findings. If a report states "cause not yet established", this must not become an established cause. Similar wording in a previous case is not sufficient evidence of comparability: different equipment, a different product or different cleaning conditions may limit its applicability. The assistant should make these differences visible; assessment by qualified personnel remains the responsibility of the designated staff.

To which situation does the information apply? The site, equipment and event period must be established for the deviation investigation assistant. Reconstruction of the cleaning cycle requires the SOP version in force at that time. If the SOP has since changed, the assistant must distinguish the versions. Current requirements may be relevant to investigation steps or actions to be performed today. For previous deviations, the equipment condition at the time and the documented investigation status must also remain clear.

Chapter 4 of the EU GMP guidelines, Documentation, provides, among other things, for clear identification, approval and updating of instruction documents. [9] The assistant should therefore identify the document and version unambiguously and clearly distinguish the original source from the generated factual summary. How the output is subsequently used determines the necessary labelling, review and approval.

Which equipment, products and cleaning types are covered? If the assistant is to support several equipment items, products or cleaning types, such as intermediate cleaning within a campaign and full cleaning at a product changeover, the availability of the necessary documents and relevant deviation types needs to be checked. Examples include interrupted cycles, process parameters outside the expected range and missing records. Situations not covered must be taken into account when defining the limits of use. [6]

Figure 3 summarises the three questions concerning the assistant's data basis.

Figure 3: The data basis of the hypothetical deviation investigation assistant is assessed in terms of content, context and coverage. Our own illustration; documentation reference: EU GMP Chapter 4 [9].

This provides a basis for checking whether the assistant meets the defined requirements in the intended workflow.

Assessing the fitness of the application system

In the deviation investigation assistant, the document collection, retrieval, model, settings and user interface work together. Their performance must be assessed in the intended workflow; testing the model in isolation is not sufficient. [1, 2]

Here, grounding means anchoring an answer in specific evidence. RAG helps provide that evidence but guarantees neither completeness nor correct conclusions. A document may be missed during retrieval, or an exception may be lost in summarisation. A cited source passage must therefore be accessible and must actually support the statement. [2, 3]

For purchased AI components, it is important to clarify early which development and test evidence the supplier provides and to what extent it covers the company's own intended use. Annex 11 provides for risk-based supplier assessment and review of the documentation supplied. [4] For the AI application, we recommend also considering available information on training data and model limitations. Application testing performed by the user does not automatically compensate for missing development evidence; remaining gaps may restrict the permitted use.

Within its scope, the draft Annex 22 describes representative and independent test data, performance metrics and documented acceptance criteria. [6] Appropriate criteria for the deviation investigation assistant need to be derived from its tasks and potential errors.

For models developed or adapted in-house, we recommend a final evaluation using data that have influenced neither training nor model selection. Similarly, after prompts and retrieval have been optimised for the assistant, held-out cases must demonstrate whether the application handles typical situations and relevant errors. [6]

Test cases include the relevant equipment, products and cleaning types, different record templates, and different types of deviations and investigation statuses. Previous cases with similar wording but substantively important differences also belong in the evaluation. Gaps in this coverage need to be documented and taken into account when defining the limits of use. [6]

A study by Liu et al. (2024) showed that the position of relevant information in the context influenced the performance of the language models investigated. The assistant should therefore also be tested with long documents, dispersed information and easily overlooked exceptions. [3]

Expert-reviewed reference results need to be established in advance for the cases in Table 1: which sources and facts must appear, which differences limit the comparability of previous cases, and which discrepancies or open questions must be identified? A suspected cause must not be presented as confirmed. Missing evidence must neither appear to exist nor be replaced by plausible assumptions.

The expected results should address retrieval and summarisation separately: was a relevant logbook entry not retrieved, or was it available to the model but omitted from the output? A suitable boundary case combines an early hypothesis about the cause with a later entry that explicitly does not confirm it. The summary should preserve the chronology and the open investigation status. Testing should also establish whether, when documents are missing or unreadable, the assistant makes the limitation visible rather than suggesting completeness.

Making human review effective

The first step is to define where people intervene. In a human-in-the-loop (HITL) arrangement, the workflow waits for human review and approval of the decision concerned. In a human-on-the-loop (HOTL) arrangement, the system acts within defined limits; people supervise and can intervene. One consideration we take from the conference discussions is that, if a decision requires tacit knowledge that is not captured in the available data, this supports review by a qualified person before the decision is made. Even where sufficient data are available, the consequences of errors, their detectability and the opportunities to intervene determine the oversight required. The EMA workshop report explicitly discusses how the performance of human oversight can be assessed. [7] We therefore recommend demonstrating its contribution to error detection under realistic working conditions for the deviation investigation assistant.

Reviewers need access to the relevant original sources, expertise, sufficient time and actual authority to intervene. [7] The user interface also influences the control. A practical question we take from the conference discussions is whether rejecting or correcting an AI output creates unnecessary additional work. For the assistant, this means that source passages must be easy to access and correction or rejection must be possible without unnecessary barriers.

The EMA workshop report addresses automation bias, meaning excessive reliance on automated outputs. The discussion emphasises that oversight must be effective and cannot compensate for inadequate validation. [7] We recommend evaluating human review of the assistant in a controlled exercise: users assess correct and deliberately incorrect outputs against the original documents. These include previous cases applied incorrectly, causes presented as confirmed without justification and omitted contradictory entries. Critical errors detected, critical errors remaining and review time are recorded under realistic conditions. This makes the actual contribution of human oversight visible.

Figure 4 connects the conditions needed for human review with its controlled evaluation.

Figure 4: Effective human review requires appropriate conditions and a controlled evaluation. Errors detected, errors remaining and review time are recorded. Our own schematic illustration based on the EMA workshop report [7].

Testing technical safeguards for their intended purpose

Technical safeguards, often called guardrails, are intended to prevent errors, detect them or limit their consequences. Input checks, change control, periodic evaluation and access restrictions are already established in Annex 11. [4] For each control, we recommend defining the error it addresses, the conditions of use and the evidence of effectiveness. Where a second model is used to check outputs, common causes of error also need to be investigated; using another model alone does not demonstrate an independent control.

Indirect prompt injection occurs when a model treats instructions in retrieved content as an instruction to act. The publicly documented EchoLeak example (CVE-2025-32711) illustrates the risk: under certain conditions, a specially crafted email could cause Microsoft 365 Copilot to disclose internal data. Microsoft has fixed the vulnerability. [15] For the assistant, testing should establish, for example, whether an injected instruction in a comments field causes it to conceal a contradictory logbook entry or falsely present a cause as confirmed. Retrieved content must not alter the defined task or technical access boundaries.

A fixed output schema for the assistant can require equipment and deviation identifiers, document ID, version or revision status, source location and open questions. Observed facts, documented hypotheses and confirmed findings are reported separately. Schema checks can detect missing mandatory fields or invalid values, for example. Assessment by qualified personnel remains necessary: even a source location given in the correct format may refer to a different event. [13]

In the example, we therefore combine schema checks with comparison against the relevant source documents and records, and verification of the cited passages. Missing or conflicting information must remain visible as open points. The responsible personnel clarify and assess them through the controlled investigation process. Read-only access to sources is technically enforced; write access to GMP records and changes to investigation, cleaning or release status are excluded. Any later transfer of reviewed content into the deviation report is performed by the responsible personnel according to the defined procedure. Preventive, detective and containment controls thus work together. [4, 7]

Keeping an AI application under control in operation

Annex 11 addresses changes, periodic evaluation, incidents and business continuity; for external service providers, it provides for agreements with clear responsibilities. [4] When AI is used, traceable versions and settings, advance notification of supplier changes, and responses to degraded performance or failure therefore need to be defined.

Conversation history, retrieved RAG passages or a memory function can change responses without changing model weights. [11, 14] Fine-tuning, retraining and online learning, by contrast, change the model itself. [6, 11] A static model therefore does not make the application system unchanging.

Figure 5 shows the two levels of change and their joint assessment under change control.

Figure 5: Changes to the application system and to model weights need to be recorded separately and assessed for their impact on the intended use. Supplier updates can affect both levels. Our own schematic illustration based on [4, 6, 11, 14].

Changes to context, memory and model version need to be recorded separately. Stored information requires rules on provenance, validity, access, correction and deletion. It must not override a valid SOP without review. Where weights change, an assessment is needed to determine which evidence of fitness for intended use must be renewed. [4] We also recommend identifying whether data subsequently used for training and testing originate from a process influenced by AI. This influence should be taken into account in data selection and independent evaluation.

For the assistant, new SOP versions, record templates, retrieval logic, prompts or access rights can affect existing evidence. After an SOP change, it must still associate each cleaning cycle with the version in force at the time of the event. Template changes must not result in comments or contradictory entries being overlooked. Changes in investigation status must also be reflected correctly: a previous case that has since been closed must not be presented using an outdated hypothesis about its cause. Metrics, review intervals and responsibilities make changes visible. Criteria also need to be defined for restricting or stopping use and for continuing the process when the system is unavailable.

For audits and inspections, the task, evidence of fitness for intended use and responsibilities must be traceable: who assesses errors, approves changes or stops use? Accountability remains with the company and the responsible personnel. [7]

Governing AI use and equipping staff

An inventory of the AI applications actually used should also include shadow AI: tools used in day-to-day work without a controlled introduction. A hypothetical example is a QA employee copying a complaint into an AI tool that has not been approved for those data, in order to understand it better. This can create an undocumented decision aid and disclose confidential information. Companies therefore need clear responsibilities and understandable rules on which systems may be used for which tasks and data. [10]

Training should reflect the tasks and decision-making authority of each role. Critical thinking can be practised using the company's own cases: which source supports the statement, what information is missing, and what would be the consequences of an incorrect recommendation? Staff must know when to discard outputs, seek clarification or escalate.

We recommend first developing expertise through manageable cases and assessing learning through relevant professional tasks. New functions or identified knowledge gaps provide reasons to update training. In this way, competence and oversight develop throughout the lifecycle. [7]

Preparing to use an AI application

The first step is a shared task description developed by process owners, quality assurance, IT and, where appropriate, data scientists. They clarify which information is needed, which errors matter and whether the available evidence supports the use. Table 2 provides a starting point for this discussion.

Table 2 Questions and possible deliverables for defining the scope of use

A pilot phase evaluates the value the AI application offers compared with the existing workflow. The comparison uses the same task and the same quality requirements. For the assistant, one relevant measure is the time taken to produce a factual summary reviewed by qualified personnel, including any corrections required. Other measures include errors by type and significance, as well as review effort and operational overhead. The MHRA case described at the beginning shows how much incorrect AI outputs can increase review effort. Acceptance criteria are defined in advance for this task.

If relevant source data are missing, critical errors remain undetected or supplier changes are not sufficiently traceable, use should be restricted or deferred. The assistant could initially be limited to making relevant documents and previous cases retrievable together with their metadata; even this narrower scope requires appropriate evidence. Our takeaway from the presentations and discussions at the conference is to understand the process and data problem first, and then choose the simplest solution that meets the requirements. That may mean better data management, conventional monitoring or rule-based automation. AI is an option where it offers additional, demonstrable value.

Basis of the article

This article draws on our attendance at the ECA Annex 22 / AI Conference in Copenhagen on 7 and 8 October 2026, our own notes, and publicly accessible regulatory and technical sources. The conference insights reflect our professional interpretation. The deviation investigation assistant in the cleaning context and its test cases are hypothetical; Catalent's Qai provides a publicly described practical reference. Our test cases and controls do not describe an investigation or validation of the Catalent system. The figures are our own schematic illustrations.

References

The numbered references are publicly accessible and linked.

[1] Attention Is All You Need. Vaswani et al., 2017, arXiv v7.

[2] Retrieval Augmented Generation for Knowledge Intensive NLP Tasks. Lewis et al., 2020, arXiv v4 (2021).

[3] Lost in the Middle How Language Models Use Long Contexts. Liu et al., 2024. Transactions of the Association for Computational Linguistics 12, pp. 157–173.

[4] EudraLex Volume 4 Annex 11 Computerised Systems. European Commission, Revision 1, January 2011. Official status page

[5] ICH Q9 R1 Quality Risk Management. ICH, Step 4 version dated 18 January 2023, corrected on 15 January 2025. Official status page

[6] Annex 22 Artificial Intelligence. European Commission, 2025 consultation draft. Consultation 7 July–7 October 2025. Official status page

[7] Good manufacturing practice: Multistakeholder workshop on expert contributions to artificial intelligence guidance development (Annex 22). EMA, Workshop held on 30 June–1 July 2026. Workshop report EMA/156789/2026, first published on 2 October 2026.

[8] Use of AI for GXP inspection responses setting standards without stifling innovation. Peter Brown, MHRA Inspectorate, 29.06.2026.

[9] EudraLex Volume 4 Chapter 4 Documentation. European Commission, Revision 1, January 2011.

[10] Generative KI-Modelle: Chancen und Risiken für Industrie und Behörden. German Federal Office for Information Security, Version 2.0, 17 January 2025.

[11] Language Models are Few-Shot Learners. Brown et al., 2020, arXiv v4.

[12] Content generation parameters. Google Cloud. Technical documentation, checked on 15 October 2026.

[13] Structured outputs. Google AI for Developers. Technical documentation, checked on 15 October 2026.

[14] Memory overview. LangChain. Technical documentation, checked on 15 October 2026.

[15] AI Application Security Series 1: Security considerations when adopting AI tools. Microsoft Detection and Response Team, 17.12.2025. Section on EchoLeak (CVE-2025-32711).

[16] Catalent: Inside Qai™: How Catalent is Teaching AI to Think Like a Quality Expert. 10.09.2026.

[17] Microsoft Customer Stories: Catalent uses AI on Azure to strengthen quality management system processes. 09.09.2026.

[18] Syntegon: Syntegon validates first AI-equipped visual inspection system in production line. Press release, accessed on 16 October 2026.

[19] Deep learning. Yann LeCun, Yoshua Bengio and Geoffrey Hinton, 2015. Nature 521, pp. 436–444. DOI: 10.1038/nature14539. Publicly accessible author manuscript.

[20] Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1, January 2023. DOI: 10.6028/NIST.AI.100-1.