Model drift

Overview

Models are trained on a specific dataset. As time passes, models may be requested to run inference on data that differs from this dataset. Also, the data the model was trained on may change. This makes model predictions less accurate over time which causes a data drift: the average data processed by the model significantly differs from the training dataset. The model’s evaluation metrics start to score lower.

When this happens, a new version of the model, trained on a new dataset, should replace the one before. Models can also be retrained on missing or newer samples to improve their predictive power. Model Ops monitors model performance metrics and alerts when retraining should be considered.

By monitoring the distribution of data feeding a model and detecting when it changes, you can learn if predictions are different than normal, faster. You can identify why models are making wrong predictions: if the models are getting wrong data or simply are no longer up to date.

Monitor models

To see model drift data, send a request to get a drift summary. This endpoint includes baseline and current distribution values with their chiSquaredValue and pValue scores.

Retrain

Update your model’s baseline with a revised dataset to minimize model drift.

Drift parameters

The list explains the drift parameters returned by Modzy.

baseline-distribution

The baseline training data used to evaluate model drift.

baseline-period

The date range that holds the training data used to evaluate model drift.

distribution

The result data values and their amount.

chiSquaredValue

Quantifies the difference between two statistical distributions. Modzy uses Pearson’s version to identify variations in model outputs over time as a drift measure. This value is unbounded.

pValue

A value between 0 and 1 related to the chiSquaredValue. It’s gets larger as the compared distributions become more dissimilar. The threshold values are compared to this value.

This value is different from the generally accepted statistical definition of the pValue, which typically decreases to get smaller as fewer distributions are available that would have a higher Chi Square value metric emerge when compared to the baseline distribution. The version Modzy uses is equal to 1 – pValue if you consider the pValue to be defined in the standard statistical way.

thresholds

Threshold values range from 0 to 1 and correspond to pValues for a Chi Squared test. It’s gets larger as the compared distributions become more dissimilar.