Bern
Lisbon
New York
How do you monitor AI after launch?
To monitor AI after launch, track four things for each use case: errors, complaints, drift in output quality and the results of regular spot checks. A tool that worked well in the pilot can get worse later, because the model, the data or the way people use it changes while the approval stays the same.
Hypothetical example: an online shop tests an AI assistant for customer questions about returns before launch. Six months later the vendor switches to a new model version, and the assistant starts quoting a return period the shop no longer offers. Nobody notices until customers complain.
What to track
- Errors. Wrong, incomplete or inappropriate outputs. Rather than logging every minor edit, define in advance which classes of error are significant (for example a wrong price, a false promise to a customer or personal data shown to the wrong person), the error rate you will watch, and the critical events that pause the tool immediately.
- Complaints. From customers or staff, about the AI tool or a process it supports. Tag them so they can be counted per use case.
- Quality drift. A gradual decline in quality, often after changes in the model, the input data or the kind of requests. Drift is hard to see day to day, so compare recent results with the pilot.
- Spot checks. A person reviews a sample of outputs at fixed intervals against the criteria used at approval. Sample size and frequency depend on the volume and on the consequences of errors: 20 checks a month may suit a low-volume, low-risk internal tool, but not a customer-facing one handling thousands of requests.
When human reviewers keep rejecting or changing the output, treat that as a signal too and find out why.
Recommendation. Give each use case an owner who reviews these signals at a set interval, decides whether anything must change and records the decision. Monitoring that nobody reads is not monitoring.
When to reassess a use case
Some changes are significant enough that the original AI risk assessment no longer holds. Run a new assessment when any of these happens:
- a new model or model version, including one the vendor rolls out automatically;
- new data, such as a new source or type of personal data;
- a new purpose or audience, for example moving an internal tool to customers;
- a new market;
- an incident, which also calls for the steps in responding to an AI incident;
- a change in regulation that affects the use case.
When you must monitor AI by law
Changes can also affect your company’s legal position, so check them against whether you are a provider or a deployer.
Legal requirement (EU AI Act, Art. 25, 26 and 72). These rules concern high-risk AI systems under Annex III and Annex I Section A; Annex I Section B products follow the sector route in Art. 2(2). A deployer or other party that substantially modifies a high-risk AI system, or changes a system’s intended purpose so that it becomes high-risk, becomes its provider and takes on the provider’s obligations (AI Act, Art. 25). A substantial modification is a change, not planned by the provider, that affects the system’s compliance or its intended purpose (Art. 3). Deployers must monitor the operation of a high-risk system on the basis of its instructions for use and inform the provider and the relevant authorities about risks and serious incidents (Art. 26(5)). Providers must run post-market monitoring: they track and document how the system performs once it is in use (Art. 72).
These duties do not share one start date (Digital Omnibus on AI):
| Obligation | Where | Applies from |
|---|---|---|
| Role change to provider | Art. 25, Chapter III, Section 3 | 2 December 2027 (Annex III) / 2 August 2028 (Annex I Section A; Section B follows the sector route in Art. 2(2)) |
| Deployer monitoring and reporting | Art. 26, Chapter III, Section 3 | same |
| Post-market monitoring (provider) | Art. 72, Chapter IX | not expressly postponed; depends on classification and the Art. 111 transitional rules |
Timing of Art. 72 for a specific system needs a legal assessment. The full timeline is in what the EU AI Act requires.
Outside the high-risk regime. Ongoing monitoring remains good practice and may also be required by other applicable laws or contracts. For example, where a tool processes personal data, GDPR Art. 32 already requires security appropriate to the risk, including regularly testing and evaluating the effectiveness of the measures (GDPR, Art. 32). The voluntary NIST AI Risk Management Framework treats managing AI risk as work that continues after deployment.
Next step: in your AI inventory, set a first review date for each use case and name the person who will carry it out.
Sources and further reading
AI Horizon Conference
The AI Horizon Conference returns to Lisbon, once again bringing together entrepreneurs, investors and industry leaders to discuss the future of AI.
