MLflow and Datadog for AI Drift Monitoring and Compliance

Q: Which compliance frameworks require continuous AI model monitoring?

Multiple frameworks require ongoing AI monitoring. The EU AI Act (Articles 12 and 72) mandates automatic logging and post-market monitoring for high-risk systems. ISO 42001 Clause 9 requires performance evaluation and monitoring. ISO 27001 requires monitoring and measurement of information security controls. NIST AI RMF's Measure function requires ongoing performance assessment. SOC 2 Trust Services Criteria require continuous monitoring of controls. The EU MDR requires post-market surveillance including performance monitoring for AI-enabled medical devices. A single well-configured monitoring stack can generate evidence for all of these simultaneously.

Model drift is not a theoretical risk. It is the default state of any AI system operating on real-world data. Training distributions shift. User behaviour changes. Upstream data pipelines introduce silent schema changes. A model that passed every validation metric at deployment can quietly degrade to the point where its outputs are unreliable, unfair, or both.

This is why every serious governance framework requires continuous monitoring. Not a one-time validation at deployment. Ongoing, measurable, documented observation of how the system actually performs in production.

The core thesis: a single MLflow and Datadog configuration, designed around drift detection, generates compliance evidence for multiple frameworks simultaneously. You do not need separate monitoring for each regulation. You need one monitoring system configured correctly.

Why Every Framework Converges on Continuous Monitoring

The requirement appears in different language across frameworks, but the obligation is structurally identical: you must continuously observe your AI system's behaviour in production and act when it degrades.

Framework	Clause / Article	Monitoring Requirement
EU AI Act	Articles 12, 72	Automatic logging for traceability; post-market monitoring system for high-risk AI
ISO 42001	Clause 9	Performance evaluation: monitoring, measurement, analysis, and evaluation of the AI management system
ISO 27001	Clause 9.1	Monitoring and measurement of information security controls, including AI system controls
NIST AI RMF	Measure function	Ongoing assessment of AI system performance, including metrics for accuracy, fairness, and reliability
SOC 2	CC7.1, CC7.2	Continuous monitoring of system operations, detection of anomalies and deviations from expected performance
EU MDR	Article 83, Annex XIV	Post-market surveillance: systematic data collection and analysis of device performance throughout its lifetime

The convergence is not a coincidence. Every framework is solving the same underlying problem: AI systems are not static. A deployed model is not finished software. It is a living system whose behaviour changes as the world around it changes. The frameworks differ in terminology and scope, but the operational requirement is the same: detect degradation, document it, and respond.

MLflow: Baselines, Experiments, and Drift-Aware Model Management

MLflow tracks experiments, runs, models, and datasets. For drift monitoring, MLflow serves as the source of truth for baselines: what the model looked like when it was validated, what data it was trained on, and what metrics it achieved at approval time. Every drift alert in Datadog is measured against an MLflow baseline.

Experiment Naming Convention

Name MLflow experiments with enough structure that a reviewer from any framework context can understand what changed, why, and what triggered the change:

# Avoid:
experiment_47
test_run_new_data

# Prefer:
credit-scoring-v2/baseline-2026-04
credit-scoring-v2/drift-investigation-input-shift-2026-06
hiring-model-v3/robustness-test-adversarial
hiring-model-v3/post-market-retrain-q1-2026

Required MLflow Run Tags for Drift Context

Add standard tags to every production-relevant MLflow run. These tags make it possible to filter runs by compliance context and generate change records across frameworks:

mlflow.set_tags({ "compliance.trigger": "drift_detected", # or: scheduled_retrain / data_refresh / incident "compliance.frameworks": "eu_ai_act,iso42001,nist_ai_rmf", # which frameworks this run addresses "drift.type": "accuracy_degradation", # or: input_distribution / fairness_shift / concept_drift "drift.severity": "medium", # low / medium / high / critical "drift.detected_date": "2026-06-12", # when the drift was first detected "data.version": "dataset-v4.1", # traceable dataset reference "data.governance_approved": "true", # data governance evidence "model.risk_assessment": "risk-2026-04-15",# links to risk assessment "reviewer": "[email protected]" # human sign-off evidence })

The drift.* tags are the key addition. When a model is retrained because of detected drift, these tags create a traceable chain from the Datadog alert that detected the drift, through the investigation, to the retraining run that resolved it. This chain satisfies ISO 42001 Clause 10 (improvement), NIST AI RMF's Manage function, and EU AI Act Article 72 post-market monitoring obligations simultaneously.

Model Registry Stage Lifecycle as a Compliance Gate

Use MLflow's model registry staging lifecycle to enforce a compliance gate before production deployment. This gate is not framework-specific; it enforces requirements that are common across all frameworks:

Staging (None) --> Staging --> Production --> Archived

# Required before transitioning Staging -> Production:
# 1. Accuracy metrics meet defined thresholds (logged as run metrics)
# 2. Bias/fairness tests pass (logged as run metrics)
# 3. Drift baselines recorded (accuracy, input distribution, fairness)
# 4. Risk assessment updated (tag: compliance.risk_assessment)
# 5. Human reviewer sign-off recorded (tag: reviewer)

The model registry transition timestamp, the approving user, and the run metrics form a verifiable record that performance, fairness, and oversight obligations were satisfied before the model went to production. This evidence applies to EU AI Act Articles 9, 14, and 15; ISO 42001 Clause 8 (operational planning and control); NIST AI RMF Map and Measure functions; and SOC 2 change management controls.

Recording Drift Baselines at Registration

When a model is registered for production, record the baseline metrics that Datadog will monitor against. These baselines are what make drift detection meaningful:

mlflow.log_params({
    "baseline.accuracy": 0.94,
    "baseline.f1_score": 0.91,
    "baseline.demographic_parity_diff": 0.03,
    "baseline.input_feature_means": "logged_as_artifact",
    "baseline.input_feature_stds": "logged_as_artifact",
    "drift.accuracy_threshold": 0.05,          # alert if accuracy drops by more than 5%
    "drift.fairness_threshold": 0.02,          # alert if fairness metric shifts by more than 2%
    "drift.input_psi_threshold": 0.2           # alert if Population Stability Index exceeds 0.2
})

Logging Dataset Lineage

Data governance requirements appear in the EU AI Act (Article 10), ISO 42001 (Clause 7.5), and NIST AI RMF (Map function). Log dataset metadata with every training run:

mlflow.log_params({
    "dataset.train.source": "s3://data-lake/training/v4.1/",
    "dataset.train.version": "4.1",
    "dataset.train.size": 1250000,
    "dataset.train.date_collected_from": "2024-01-01",
    "dataset.train.date_collected_to": "2025-12-31",
    "dataset.train.geographic_scope": "EU",
    "dataset.train.demographic_groups_represented": "documented_in_data_card",
    "dataset.validation.source": "s3://data-lake/validation/v4.1/",
    "dataset.test.source": "s3://data-lake/test/v4.1/"
})

Datadog: Configuring Monitoring for Drift Detection

Datadog is where live drift detection happens. The goal is twofold: detect when model behaviour deviates from MLflow baselines, and structure the resulting logs so they constitute compliance evidence across frameworks.

Inference Log Schema for Multi-Framework Evidence

Each log event for an AI inference should contain enough context to satisfy monitoring requirements from any framework. At minimum:

{
  "timestamp": "2026-06-15T09:14:22.341Z",   # UTC, millisecond precision
  "system_id": "hiring-screener-v3",          # unique identifier for the AI system
  "model_version": "3.2.1",                   # must match MLflow registered model version
  "request_id": "req_7f8a91bc",              # unique per inference request
  "input_reference": "sha256:a3f8...",        # hash of input, not raw input (privacy)
  "output_class": "SHORTLIST",               # the decision made
  "confidence_score": 0.87,
  "human_override": false,                   # was this overridden by a human reviewer?
  "oversight_flag": false,                   # was this flagged for human review?
  "session_id": "sess_4a2b19",              # groups requests per use session
  "compliance.monitored": true               # explicit tag for compliance filtering
}

Key design decisions:

Hash inputs, do not log raw personal data. GDPR data minimisation applies regardless of which framework triggered the monitoring. A hash proves the input was a specific value without storing the value itself
Log human override and oversight flag. These fields generate evidence for EU AI Act Article 14 (human oversight), ISO 42001 Clause 6.1.2 (risk treatment), and NIST AI RMF Govern function (human-AI interaction)
Tag with compliance.monitored. This makes it straightforward to generate compliance-filtered exports for any auditor, regardless of which framework they are auditing against

Drift Detection Monitors

Name Datadog monitors to make their drift detection purpose and framework context explicit. When a monitor fires, the name should tell the on-call engineer what drifted, how severely, and which compliance obligations are affected:

# Avoid:
"Model accuracy alert"
"High error rate"

# Prefer:
"[DRIFT] Hiring model accuracy degradation >5% (7-day rolling)"
"[DRIFT] Credit model demographic parity shift detected"
"[DRIFT] Input distribution PSI >0.2 on credit-scoring-v2"
"[MONITOR] Inference logging coverage <100% -- missing events"
"[MONITOR] Human review queue backlog >24h -- oversight SLA breach"
"[MONITOR] Data distribution shift detected -- investigation required"

Each of these monitors generates evidence for multiple frameworks. An accuracy degradation alert satisfies EU AI Act Article 72 (post-market monitoring), ISO 42001 Clause 9.1 (performance monitoring), NIST AI RMF Measure (ongoing assessment), and SOC 2 CC7.2 (anomaly detection). One alert, multiple compliance records.

Log Retention Configuration

Retention requirements vary by framework. The EU AI Act (Article 18) requires 10-year retention. ISO 42001 and ISO 27001 require retention periods defined in your records management policy. SOC 2 typically requires evidence for the audit period (usually 12 months). EU MDR requires retention for the lifetime of the device plus 10 years.

The practical approach: configure for the longest applicable retention period and use tiered storage to manage cost.

Create a dedicated Datadog Log Archive for compliance-tagged events (compliance.monitored:true)
Route this archive to long-term storage (S3, Azure Blob) with a retention policy matching your longest obligation
Separate compliance logs from operational logs to avoid incurring full Datadog ingestion costs on archived records
Document the archive configuration in your monitoring plan (referenced by your ISO 42001 Clause 9 documentation, EU AI Act Annex IV Section 7, and NIST AI RMF Measure documentation)

Generating Compliance Exports

When an auditor requests evidence, you need to produce a compliance log export quickly, regardless of which framework they are auditing against. Set up saved Datadog queries for each export type:

# Saved Datadog Log Query: "Compliance Monitoring Export"
compliance.monitored:true system_id:hiring-screener-v3
  @timestamp:[2026-01-01 TO 2026-07-23]
  | fields timestamp, system_id, model_version, request_id,
             output_class, human_override, oversight_flag
  | sort by timestamp asc

The Drift Detection Evidence Chain

When MLflow and Datadog are configured this way, every drift event produces a traceable chain that satisfies multiple frameworks:

Datadog detects that production metrics have diverged from MLflow baselines
The drift alert triggers an investigation, documented as an MLflow experiment run with drift.* tags
The MLflow run record shows the baseline metrics, the detected deviation, and the root cause analysis
If retraining is required, the new model passes through the registry compliance gate before production deployment
The model version in Datadog inference logs matches the registered model version in MLflow, closing the loop

This chain is not a compliance artefact you produce for audits. It is the normal operational record of a well-monitored AI system. The same chain satisfies an EU AI Act market surveillance authority, an ISO 42001 surveillance auditor, a SOC 2 auditor reviewing continuous monitoring controls, and a NIST AI RMF assessment. One system, one configuration, multiple frameworks.

AI systems drift. Regulation punishes undetected drift. The engineering teams that configure MLflow and Datadog for drift detection are not just building better models. They are building compliance evidence as a byproduct of good engineering practice.

Frequently Asked Questions

Which compliance frameworks require continuous AI model monitoring?

Multiple frameworks converge on the same requirement. The EU AI Act (Articles 12 and 72) mandates automatic logging and post-market monitoring for high-risk systems. ISO 42001 Clause 9 requires performance evaluation and monitoring. ISO 27001 requires monitoring and measurement of information security controls. NIST AI RMF's Measure function requires ongoing performance assessment. SOC 2 Trust Services Criteria require continuous monitoring of controls. The EU MDR requires post-market surveillance including performance monitoring for AI-enabled medical devices. A single well-configured monitoring stack can generate evidence for all of these simultaneously.

How do MLflow and Datadog work together for drift detection?

MLflow tracks the baseline: training data distributions, model metrics at validation time, and approved model versions in the registry. Datadog tracks the live system: inference logs, production metrics, and real-time alerts. Drift detection works by comparing Datadog production metrics against MLflow baselines. When Datadog detects that accuracy, fairness, or input distributions have shifted beyond thresholds defined in MLflow run parameters, it triggers alerts that feed back into the MLflow experiment cycle for investigation and retraining.