Model drift after an EHR version upgrade

1. System failure identification

   A predictive model degrades after a routine records upgrade. A field was renamed, a default changed, and the model continues scoring against inputs whose meaning has shifted underneath it. The predictions remain plausible, which is what delays discovery.

2. Data pipeline diagnostics

   Upgrades are validated for application function, not for downstream analytical consumers. No dependency map links a schema field to the models that read it, so nothing flags the break. The dependency exists in practice and is documented nowhere.

3. Predictive computational model

   Monitor input distributions per model: compare live feature distributions against the training window and alarm on divergence, independently of whether the predictions still look plausible. Distribution monitoring catches the break without needing ground-truth outcomes.

4. Workflow integration

   Add downstream model validation to the upgrade checklist as a release gate. The right integration point is the change process, not a runtime alert after the fact. A gate before release is cheaper than detection after deployment.

Back to the case study database