🎯 Monitoring & Optimisation — Objectives¶

🎯 Objective¶

The primary objective of the Monitoring & Optimisation phase is to safeguard the performance, ethical integrity, and cost efficiency of the AI system throughout its entire operational lifespan. An AI system does not stop evolving when it goes live — data distributions shift, user behaviour changes, and external conditions evolve. Without active monitoring, the system degrades silently, eroding trust and value.

This phase establishes a continuous cycle of observation, analysis, and improvement. We monitor technical performance (latency, error rates, uptime), behavioural performance (accuracy, drift, fairness), and economic performance (cost per transaction, ROI). We feed production insights back into the development environment for analysis and improvement. And we maintain the compliance posture through post-market surveillance and audit-ready logging.

Critically, we also define in advance when the system should be decommissioned. An AI system has a finite lifespan — knowing when to stop is as important as knowing when to start.

Key result: A stable, self-correcting AI ecosystem that continues to deliver demonstrable business value, remains compliant with legislation, is optimised for cost and sustainability, and has a defined decommissioning plan with clear triggers.

✅ Entry Criteria (Definition of Ready)¶

Before this phase starts, the following conditions must be met:

• The system is live with Gate 4 (Go-live) approved.
• Monitoring dashboards and alerts are active and receiving data from the production environment.
• The operations team (Operations/MLOps) is instructed, has runbooks for common scenarios, and is on standby.
• The Incident Response Plan has been tested with at least one simulation exercise.
• Baseline metrics are recorded from the Go-live configuration — these serve as the reference point for drift detection.

⚙️ Core Activities¶

1. Performance Monitoring¶

We monitor the technical and behavioural "heartbeat" of the system. This is the continuous observation that detects degradation before it becomes an incident.

• Real-time Metrics: Dashboarding of critical metrics — Latency (response speed), Error rates (failures per transaction), Uptime (availability percentage), Throughput (transactions per second).
• Drift Detection: Statistically monitoring whether production input data deviates from training data (Data Drift) or whether the relationship between data and outcomes changes (Concept Drift).
• Fairness Monitoring: Periodic sampling to detect emerging bias across demographic or business-relevant groups.

2. Continuous Improvement & Retraining¶

We use production insights to improve the system. Standing still means falling behind — the world changes, and the AI must change with it.

• Retraining Strategy: Define when to retrain — periodically (e.g., quarterly), on drift alert, or on new data threshold. The strategy depends on the risk level and the rate of change in the domain.
• Experiment Loops: Test new hypotheses in short sprints using production data. Run A/B tests or Canary releases to validate improvements before full deployment.
• Backlog Management: Maintain a living list of bugs, improvements, and feature requests from users. Prioritise based on impact and effort.

3. Cost Control & Energy Efficiency¶

We optimise the economic and ecological footprint of the AI system. Sustainability is measured in both euros and CO₂.

• Cloud & API Optimisation: Monthly review of compute (GPU/CPU) and token costs. Optimise through model compression (quantisation), caching, or architectural changes.
• Sustainability Measurement (ESG): Monitor energy consumption (inference footprint) and report for ESG goals. Refer to Green AI benchmarks for comparison.
• Resource Allocation: Set up autoscaling to adjust infrastructure to actual demand. Avoid over-provisioning (waste) and under-provisioning (degraded performance).

4. Ethical Oversight & Compliance Monitoring¶

We maintain the compliance posture through ongoing surveillance. The EU AI Act requires post-market surveillance for high-risk systems — but the principle applies to all risk levels.

• Post-Market Surveillance: Continuously scanning for unforeseen bias, discrimination, or safety risks. The Guardian performs periodic reviews.
• Audit-ready Logging: Retaining logs of decisions and human interventions for auditors. The logging requirements depend on the risk level.
• Transparency Reports: Periodic reporting to stakeholders and CAIO on safety, performance, and compliance.
• Fairness Audit (Bias Audit): Regular sampling by the Ethicist of the "tone" and quality of outputs.

5. Decommissioning Planning¶

We define in advance when shutdown is justified. An AI system has a finite lifespan — technical obsolescence, economic unviability, ethical concerns, or strategic shifts can all trigger decommissioning.

• Decommissioning Triggers: Define the conditions under which the system should be shut down (technical, economic, ethical/legal, strategic).
• Decommissioning Process: Document the steps for controlled wind-down — announcement, archiving, knowledge transfer, data deletion, infrastructure shutdown, and Guardian sign-off.

👥 RACI¶

✅ Exit Criteria (Ongoing Phase)¶

The Monitoring & Optimisation phase is continuous — it does not have a traditional exit. However, the phase is considered "healthy" when:

• Monitoring dashboards show all metrics within acceptable thresholds.
• Drift detection is active with defined alert thresholds.
• Retraining schedule is defined and adhered to.
• Cost reviews are conducted monthly.
• Fairness audits are conducted per the schedule for the risk level.
• Transparency reports are produced and distributed to stakeholders.
• Decommissioning triggers are defined and documented.

Collaboration Mode: [Mode X — Name]. Drift thresholds and human checkpoint depend on mode. Required validation for this mode: → See Evidence Standards.

📦 Deliverables¶

The following artefacts are produced and maintained during this phase:

1. Monitoring Dashboard — real-time metrics for technical, behavioural, and economic performance.
2. Drift Detection Report — periodic analysis of data drift and concept drift.
3. Retraining Log — record of retraining events, triggers, and results.
4. Cost Overview (Monthly) — updated cost analysis with optimisation recommendations.
5. Transparency Report (Quarterly) — safety, performance, and compliance summary for stakeholders.
6. Decommissioning Plan — triggers, process, and responsibilities for controlled wind-down.

Next step: Set drift thresholds and schedule the first quarterly review. → Use the Gate 4 Checklist as your starting point. → See also: Activities | Drift Detection | Continuous Improvement

Version: 1.1 Date: 07 May 2026 Status: Final