Data Governance¶

1. Data Quality Framework¶

Data quality is measured along six dimensions. Define concrete thresholds per dimension that match the risk level of the project.

2. Data Lineage & Provenance¶

What is data lineage?¶

Data lineage is the complete description of the origin, transformations and movements of data — from source to model input and ultimately model output.

Why does it matter?¶

• Traceability: When unexpected model results occur, you can quickly identify which data is the cause.
• Debugging: Identify exactly where in the pipeline a data error was introduced.
• Compliance: The EU AI Act requires that the provenance of training data is demonstrable for high-risk systems.
• Reproducibility: Without lineage you cannot reliably repeat experiments.

How to implement?¶

Minimum requirements:

• Every dataset has a unique identifier and version number
• Transformation steps are recorded with input version, output version and timestamp
• Metadata tags include: source, owner, processing date, quality score

Tooling options:

Minimum requirements per risk level:

3. Data Contracts¶

What are data contracts?¶

A data contract is a formal agreement between a data producer (the team that delivers data) and a data consumer (the team that uses data). It prevents changes in upstream data from unexpectedly breaking your AI pipeline.

Components of a data contract¶

Example contract template¶

# Data Contract — [Dataset Name]
contract_version: "1.0"
producer:
  team: "Customer Service Team"
  contact: "name@organisation.com"
consumer:
  team: "ML Platform Team"
  contact: "name@organisation.com"
dataset:
  name: "customer_interactions"
  format: "parquet"
  location: "s3://data-lake/customer_interactions/"
schema:
  - field: "customer_id"
    type: "INT"
    nullable: false
  - field: "interaction_date"
    type: "DATE"
    nullable: false
  - field: "channel"
    type: "VARCHAR(50)"
    nullable: false
    allowed_values: ["email", "phone", "chat", "portal"]
sla:
  refresh: "daily before 06:00 UTC"
  availability: "99.5%"
quality_rules:
  completeness: "≥ 98%"
  uniqueness_on: "customer_id + interaction_date"
change_policy: "Breaking changes: minimum 2 sprints advance notice via RFC"

4. Data Versioning¶

Why?¶

Without data versioning you cannot guarantee that a model training run is reproducible. If training data changes without version tracking, debugging and auditing become impossible.

Approach¶

Minimum requirements:

• Every training dataset has a unique version number or hash
• The relationship model version ↔ data version is recorded in the model registry
• Previous versions are queryable for debugging and auditing
• Changes to datasets are logged (what changed, when, by whom)

5. Metadata Management¶

Good metadata makes data findable, understandable and reusable.

Minimum metadata per dataset¶

Data catalogue¶

Tooling options: DataHub, Amundsen, Apache Atlas, Collibra, or a simple internal wiki.

6. Practical Checklist per Phase¶

Phase 1 — Discovery¶

• Data sources inventoried and documented
• Initial quality measurement performed (sample across the six dimensions)
• Data ownership established per source
• Privacy classification assigned (does the data contain PII?)
• Initial data lineage sketched (source → processing → usage)

Phase 2 — Validation¶

• Data contracts established with all relevant producers
• Automated quality controls set up in the pipeline
• Data versioning configured for training sets
• Metadata populated in the data catalogue
• Quality thresholds defined and agreed with the team

Phase 3 — Development¶

• Data contracts actively enforced (monitoring for violations)
• Full lineage tracking operational
• Quality reports automated and visible in dashboards
• Data versioning integrated with model registry
• Metadata up-to-date and searchable

Phase 4+ — Monitoring & Ongoing¶

• Continuous data quality monitoring active
• Drift detection on input data (not just model output)
• Periodic review of data contracts (at least quarterly)
• Data catalogue updated for new or modified datasets
• Audit trail available for compliance reviews

7. Related Modules¶

• Data Pipelines — technical standards for data ingestion, transformation and validation
• Data Evaluation (Phase 1) — initial data quality assessment in the Discovery phase
• Drift Detection — detection of shifts in data and model behaviour
• Data & Privacy Sheet — privacy aspects of data processing
• Evidence Standards — logging and auditability