⚙️ Monitoring & Optimisation — Activities¶

🎯 Objective¶

Execute the Monitoring & Optimisation phase activities to maintain system performance, detect and respond to drift, control costs, ensure ongoing compliance, and manage the system through its operational lifespan — including decommissioning when appropriate.

✅ Entry Criteria (Definition of Ready)¶

• System is live with Gate 4 approved.
• Monitoring dashboards and alerts are active.
• Baseline metrics are recorded from the Go-live configuration.

⚙️ Core Activities¶

1. Operational Monitoring & MLOps¶

We monitor the "heartbeat" of the system. This is the continuous observation that detects degradation before it becomes an incident.

Real-time Performance Tracking:

1. Set Up Dashboards: Configure dashboards for critical metrics:

   • Latency: Response time per request. Set alert threshold at p95 > [X] ms.
   • Error Rate: Failures per transaction. Set alert threshold at > [X]% over [Y] minutes.
   • Uptime: Availability percentage. Target: 99.9% or higher.
   • Throughput: Transactions per second. Monitor for capacity planning.

2. Configure Alerts: Set up alerts for threshold violations. Route alerts to the right people:

   • Critical alerts (system down, Hard Boundary violation): Page the on-call engineer immediately.
   • Warning alerts (performance degradation, elevated error rate): Notify the team via Slack/email.
   • Info alerts (usage spikes, cost anomalies): Log for review in the next cost meeting.

3. Establish Review Cadence:

   • Daily: Check dashboards for anomalies.
   • Weekly: Review alert history and resolve any open incidents.
   • Monthly: Produce a performance summary for the AI Product Manager and CAIO.

Performance Degradation Monitoring (Drift Detection):

1. Data Drift: Statistically monitor whether production input data deviates from training data. Use metrics like Population Stability Index (PSI) or Kolmogorov-Smirnov test. Alert when drift exceeds the defined threshold.
2. Concept Drift: Monitor whether the relationship between data and outcomes changes. This is harder to detect — it requires comparing model predictions with actual outcomes over time. A decline in accuracy or F1 score is a signal.
3. Define Significant Degradation: Performance degradation is significant if any of the following occurs relative to the baseline:
   • Factual accuracy drops ≥ 2 percentage points.
   • Relevance score drops ≥ 0.3 on a 1–5 scale.
   • Number of Major errors increases ≥ 50% over two consecutive measurement periods.

Data Loop Integration:

1. Feed Production Data Back: Set up a pipeline that feeds production data and outcomes back into the development environment. This is the feedback loop that enables continuous improvement.
2. Label Production Data: Where possible, have humans label production data to create new training examples. This is especially valuable for edge cases that were not in the original Golden Set.
3. Store for Analysis: Archive production data (with appropriate privacy protections) for drift analysis, retraining, and audit purposes.

2. Continuous Improvement & Retraining¶

Standing still means falling behind. We use production insights to improve the system through structured experiment loops.

Retraining Strategy:

1. Define Retraining Triggers:

   • Periodic: Retrain on a fixed schedule (e.g., quarterly). Suitable for stable domains with slow change.
   • On Drift Alert: Retrain when drift detection signals significant degradation. Suitable for dynamic domains.
   • On New Data Threshold: Retrain when a sufficient volume of new labelled data is available. Suitable for high-volume systems.

2. Plan the Retraining Process:

   • Assemble the new training dataset (existing data + new labelled production data).
   • Run the training pipeline and evaluate against the Golden Set.
   • Compare the new model's performance with the current production model.
   • If the new model is better, proceed through the controlled change process (specification, validation, approval).

3. Document Retraining Events: Record every retraining event in the Retraining Log: date, trigger, dataset, model version, evaluation results, and approval status.

Experiment Loops:

1. Formulate Hypotheses: Based on production insights, user feedback, or drift analysis, formulate hypotheses for improvement. Example: "Adding the new policy documents to the knowledge base will improve accuracy on compliance questions by 5%."
2. Run Experiments: Test hypotheses using A/B tests or Canary releases. A/B tests compare two versions with different user groups. Canary releases roll out the new version to a small percentage of traffic and monitor for issues.
3. Evaluate Results: Compare experiment results with the hypothesis. Did the change improve performance? Did it introduce new issues? Document the findings.
4. Decide: Promote the change to production, iterate further, or abandon the hypothesis.

Backlog Management:

1. Collect Inputs: Gather bugs, improvements, and feature requests from user feedback, monitoring alerts, and stakeholder input.
2. Prioritise: Rank items based on impact (business value, risk reduction) and effort (development time, complexity).
3. Plan Sprints: Select items for the next development cycle. Ensure the sprint capacity matches the team's velocity.
4. Track Progress: Update the backlog as items are completed, deferred, or cancelled.

3. Cost Control & Energy Efficiency¶

Sustainability in euros and CO₂. We optimise the economic and ecological footprint of the AI system.

Cloud & API Optimisation:

1. Monthly Cost Review: Review compute (GPU/CPU) and token costs. Break down by model, endpoint, and user group. Identify cost drivers and anomalies.

2. Optimisation Techniques:

   • Model Compression (Quantisation): Reduce model size and usage cost with minimal accuracy loss.
   • Caching: Cache frequent queries to avoid redundant inference.
   • Batching: Process multiple requests in a single inference call where latency permits.
   • Model Selection: Use smaller, cheaper models for tasks that do not require the full capability of the largest model.

3. Set Budget Alerts: Configure alerts when costs approach or exceed the budget. This prevents surprise bills.

Sustainability Measurement (ESG):

1. Monitor Energy Consumption: Track the usage costs — the energy consumed per transaction. Use cloud provider tools or third-party calculators.
2. Report for ESG Goals: Include AI energy consumption in the organisation's ESG reporting. Compare with Green AI benchmarks.
3. Optimise for Sustainability: Apply the same optimisation techniques (quantisation, caching, batching) to reduce energy consumption alongside cost.

Resource Allocation:

1. Set Up Autoscaling: Configure infrastructure to scale up during peak demand and scale down during low demand. This avoids over-provisioning (waste) and under-provisioning (degraded performance).
2. Monitor Resource Utilisation: Track CPU, GPU, and memory utilisation. Identify underutilised resources that can be downsized.
3. Plan Capacity: Use usage trends to forecast future capacity needs. Plan procurement or decommissioning accordingly.

4. Ethical Oversight & Compliance Monitoring¶

Ongoing legal conformity. The EU AI Act requires post-market surveillance for high-risk systems — but the principle applies to all risk levels.

Post-Market Surveillance:

1. Continuous Scanning: Monitor production output for unforeseen bias, discrimination, or safety risks. Use automated tools (content filters, toxicity detectors) and human review (Guardian sampling).
2. Incident Tracking: Log every incident — Hard Boundary violations, user complaints, fairness concerns. Track resolution status and root cause.
3. Guardian Review: The Guardian performs periodic reviews (monthly for High Risk, quarterly for Limited Risk) of the surveillance findings and recommends actions.

Audit-ready Logging:

1. Retain Logs: Keep logs of decisions and human interventions for the retention period defined by the risk level:
   • Minimal/Limited: standard 90 days, unless otherwise required.
   • High Risk: standard 12 months (or longer if legally required).
2. Ensure Queryability: Logs must be queryable by the AI PM, compliance team, and auditors. Include: date/time, user/role, use case, model version, prompt version, sources used, output, and any human override.
3. Protect Privacy: Pseudonymise personal data in logs where required by GDPR.

Transparency Reports:

1. Produce Reports: Generate periodic reports (quarterly) covering:
   • Safety: incidents, Hard Boundary violations, and resolutions.
   • Performance: accuracy, drift, and reliability metrics.
   • Compliance: audit findings, Guardian reviews, and regulatory updates.
2. Distribute to Stakeholders: Share reports with the CAIO, Business Sponsor, Guardian, and relevant stakeholders.
3. Act on Findings: Use the report to identify improvement areas and prioritise the backlog.

Fairness Audit (Bias Audit):

1. Sample Outputs: The Guardian selects a representative sample of production outputs for review.
2. Assess Tone and Quality: Evaluate whether outputs are fair, unbiased, and appropriate for the context.
3. Quantify Where Possible: If relevant groups can be distinguished, measure the difference in Major error rate between groups. For Limited Risk: ≤ 10%. For High Risk: ≤ 5%.
4. Document Findings: Record the audit results and any recommended mitigations.

5. Decommissioning¶

An AI system has a finite lifespan. Define in advance when shutdown is justified and execute a controlled wind-down.

Decommissioning Triggers:

Decommissioning Process:

1. Announcement: Inform users and stakeholders in advance (minimum 4 weeks). Explain the reason for decommissioning and the timeline.
2. Archiving: Retain the technical dossier, validation reports, and Kaizen Log per the retention policy. These may be needed for future audits or reference.
3. Knowledge Transfer: Document lessons learned in the Lessons Learned register. Share insights with other teams building AI systems.
4. Data Deletion: Delete or anonymise production data in accordance with GDPR. Document the deletion process and obtain Guardian confirmation.
5. Infrastructure Shutdown: Shut down compute instances, API keys, and monitoring pipelines. Verify that no residual costs are incurred.
6. Guardian Sign-off: The Guardian confirms that all Hard Boundaries obligations have been fulfilled, including data deletion and archival requirements.

👥 RACI¶

✅ Exit Criteria (Ongoing Phase)¶

The Monitoring phase activities are considered "healthy" when:

• Dashboards show all metrics within acceptable thresholds.
• Drift detection is active with defined alert thresholds.
• Retraining schedule is defined and the last retraining event is documented.
• Monthly cost review is completed.
• Fairness audit is completed per the schedule for the risk level.
• Decommissioning triggers are defined and documented.

Collaboration Mode: [Mode X — Name]. Drift thresholds and human checkpoint depend on mode. Required validation for this mode: → See Evidence Standards.

📦 Deliverables¶

1. Monitoring Dashboard — real-time metrics for technical, behavioural, and economic performance.
2. Drift Detection Report — periodic analysis of data drift and concept drift.
3. Retraining Log — record of retraining events, triggers, and results.
4. Cost Overview (Monthly) — updated cost analysis with optimisation recommendations.
5. Transparency Report (Quarterly) — safety, performance, and compliance summary.
6. Decommissioning Plan — triggers, process, and responsibilities.

Next step: Set drift thresholds and schedule the first quarterly review. → Use the Gate 4 Checklist as your starting point. → See also: Objectives | Drift Detection | Continuous Improvement

Version: 1.1 Date: 07 May 2026 Status: Final