MLOps Observability on Azure, AWS and Google Cloud

What does MLOps observability actually mean?

OpenTelemetry defines observability as the ability to understand a system from the outside, without knowing its internals — aimed at the "unknown unknowns" and the question "why is this happening?" — built on logs, metrics and traces. That definition transfers to machine learning, but the failure mode it has to catch is different from ordinary software.

A degrading model does not throw errors. It keeps returning well-formed, confident, wrong answers at normal latency, so every conventional signal — uptime, error rate, p99 — can sit green while accuracy collapses. The thing being monitored is not whether the service responds, but whether the statistical relationship it learned during training still holds against the data arriving today.

One honest caveat worth stating, because much of the published writing on this topic asserts otherwise: there is no authoritative standards-body definition separating ML "monitoring" from ML "observability." OpenTelemetry does not draw that contrast, and does not itself use the common "three pillars" phrasing. Treat the distinction as editorial convention rather than a specification, and judge tools by the signals they actually compute.

What are you actually monitoring?

Azure Machine Learning publishes the most complete signal taxonomy of the three clouds. Data drift compares the distribution of incoming features against the training baseline; prediction drift does the same for the model's outputs; data quality checks structural integrity rather than distribution — null value rate, data type error rate and out-of-bounds rate.

Beyond those, feature attribution drift, model performance (classification: accuracy, precision, recall; regression: MAE, MSE, RMSE) and a generative-AI generation safety and quality signal (groundedness, relevance, fluency, similarity, coherence) are all documented as preview features.

Google Cloud splits the same territory differently, and the distinction is operationally important. Vertex AI defines training-serving skew as production feature distributions deviating from the distributions used to train the model, and inference drift as production feature distributions changing significantly over time. The practical consequence: skew detection requires access to the original training dataset, drift detection does not. If the training data is gone or was never versioned, drift detection is the only option left.

Amazon SageMaker Model Monitor covers four monitor types — data quality, model quality, bias drift and feature attribution drift — with baselines computed from the training dataset by a Deequ-based engine that emits statistics (mean, standard deviation, min, max, unique count) and constraints (data type, completeness, value ranges). See the section below on its availability status before planning around it.

How does each cloud actually detect drift?

Azure ML computes a baseline distribution from training data, computes the production distribution over a lookback window, then runs a statistical test or distance score and raises an anomaly when a user-specified threshold is exceeded. Its drift signals support Jensen-Shannon Distance, Population Stability Index, Normalized Wasserstein Distance, the Two-Sample Kolmogorov-Smirnov Test and Pearson's Chi-Squared Test.

Chebyshev Distance is additionally available for prediction drift, and data quality metrics are supported to a precision of 0.00001.

Lookback windows are expressed in ISO 8601 durations (P7D, P24D, P0D), and the documented defaults are not symmetric: the production lookback window defaults to the monitoring frequency, the reference lookback window defaults to the full dataset, and the reference lookback offset defaults to twice the production window size — explicitly so there is enough reference data for a statistically meaningful comparison.

Vertex AI is far more opinionated and far less configurable. It uses L-infinity distance for categorical features and Jensen-Shannon divergence for numerical features, and ships a single default alert threshold of 0.3 applied to every categorical and numerical feature, overridable per feature. Monitoring runs every 24 hours by default with a minimum granularity of one hour (monitorInterval accepts 3600 seconds as its floor), and by default every prediction request is logged, with sampling rate configurable as a fraction.

SageMaker Model Monitor schedules monitoring jobs by cron, with an hourly minimum interval, and AWS documents that there may be a delay of 0 to 20 minutes between the requested execution time and the actual run. An analysis window's start and end must not differ by more than 24 hours. It is limited to tabular data on single-model endpoints, and data capture stops if disk utilisation passes 75 percent.

What drift metrics, thresholds and intervals does each cloud use by default?

The three clouds disagree on nearly everything measurable here — which metrics are computed, whether a default threshold exists at all, and how often monitoring can run.

Is SageMaker Model Monitor still available to new customers?

This is the single most consequential fact for anyone planning ML monitoring on AWS in 2026, and it appears verbatim on every Model Monitor documentation page: "Amazon SageMaker Model Monitor is no longer open to new customers. Existing customers can continue to use the service as normal. AWS continues to invest in security and availability improvements for Model Monitor, but we do not plan to introduce new features."

AWS does not publish a date for when the service closed, and does not state an end-of-support date — so neither should anyone else. The practical consequence is unambiguous though: a team standing up a new ML workload on AWS today cannot adopt the managed product, and should budget for assembling monitoring rather than configuring it.

What AWS points to instead is a set of open-source SageMaker AI monitoring solutions built on MLflow and Evidently AI, with QuickSight for visualisation and CloudWatch for alerting. Those reference solutions ship concrete defaults that are useful as starting points even outside AWS: alert when more than 30 percent of features drift, a drift threshold of 0.05 for the drifted-columns share, and model-quality thresholds of F1 0.70, accuracy 0.80 and ROC AUC 0.75.

Is there a standard for LLM and agent observability yet?

Partly. The vendor-neutral answer is OpenTelemetry's GenAI semantic conventions, which have moved out of the main semantic-conventions repository into their own project — but their maturity is precise and limited: the conventions are marked Development, every metric instrument is marked Development, and the schema URL in the specification is still literally "TODO."

Only the shared attributes error.type, server.port and server.address are marked Stable.

The metric instruments defined there are all histograms, and cover the shape of an LLM or agent call rather than its content: gen_ai.client.operation.duration, gen_ai.server.request.duration, gen_ai.server.time_per_output_token, gen_ai.server.time_to_first_token, gen_ai.invoke_workflow.duration, gen_ai.invoke_agent.duration, gen_ai.invoke_agent.inference_calls, gen_ai.invoke_agent.tool_calls and gen_ai.execute_tool.duration. Time-to-first-token and time-per-output-token are the two that matter most for user-perceived latency on streaming responses.

The sensible position today is to instrument against these conventions but expect breaking changes, and to avoid building anything load-bearing — dashboards, alert rules, billing attribution — on the assumption that attribute names are settled. On the managed side, Azure ML's generation safety and quality signal (groundedness, relevance, fluency, similarity, coherence) is the closest first-party equivalent, and it is still in preview.

Where does monitoring fit in an MLOps practice?

Monitoring is the stage that decides whether everything upstream was worth building, and three choices determine whether it works: version the training data so skew detection stays possible at all, set thresholds per feature rather than accepting one platform default — Vertex AI's 0.3 — across hundreds of them, and route anomalies into the same alerting path as the rest of the platform.

A model retrained on a schedule nobody validates, against data nobody profiles, degrades silently — and the first signal arrives from a customer rather than a dashboard.

That wiring is a platform decision as much as a data-science one, which is why it belongs alongside the pipeline, registry and deployment work covered in the MLOps guide rather than bolted on after a model is already in production.