You deploy a vibration anomaly model to a pump in a factory. It works great for six months. Then it starts missing failures. The pump breaks down. Production stops. You investigate and find the sensor patterns changed. The model was trained on summer data. Now it’s winter. The baseline vibration shifted. The model doesn’t recognize it.

This happens all the time. Models work in the lab. They work for a while in production. Then the world changes. Sensor distributions shift. User behavior changes. Hardware gets replaced. The model becomes blind.

This article shows how to keep edge models useful after deployment. How to detect when they’re drifting. How to retrain them. How to roll out updates safely. How to roll back when things go wrong.

The Real Problem: AI on Devices Doesn’t Age Well

Let’s start with a concrete example. A vibration sensor on an industrial pump. You train a model to detect anomalies. It learns normal vibration patterns. When something’s wrong, the pattern changes. The model flags it.

You test it in the lab. You test it on similar pumps. It works. You deploy it to fifty pumps across three factories.

Month one: perfect. Zero false alarms. Catches two real failures early.

Month three: still good. One false alarm, but it’s explainable.

Month seven: problems start. False alarms increase. Then a real failure gets missed. A pump seizes. Production line stops for eight hours.

You pull the logs. The vibration patterns changed. Not dramatically. Just enough. The model’s confidence scores dropped. But you weren’t watching. You didn’t have alerts set up. You didn’t know the model was drifting.

This is drift. The world changed. The model didn’t.

What is Drift in an AIoT Context?

Drift means the model’s assumptions about the world no longer match reality. In AIoT, there are three main types.

Data drift: The distribution of sensor inputs changes. Maybe temperature sensors read higher after a firmware update. Maybe accelerometers get replaced with different models. Maybe the environment changes—a factory gets new equipment that affects vibration patterns.

Concept drift: What counts as “normal” changes. A pump that used to vibrate at 2.5 Hz might now vibrate at 2.8 Hz and still be healthy. The hardware aged. The baseline shifted. The model thinks it’s an anomaly, but it’s not.

Hardware/infrastructure drift: Sensors get replaced. Firmware updates change how sensors report data. Calibration drifts over time. The device itself changes, but the model expects the old behavior.

Why Drift is Worse at the Edge

In the cloud, you have visibility. You see every prediction. You see every input. You can monitor in real time.

At the edge, it’s different.

Limited observability: Devices send summaries, not raw data. You might get one telemetry packet per hour. You don’t see every inference. You don’t see every sensor reading.

Irregular connectivity: Devices go offline. They’re on cellular networks with spotty coverage. They’re in basements with weak Wi-Fi. Telemetry arrives late or not at all.

Resource constraints: Devices have limited CPU, memory, and battery. You can’t run heavy monitoring on device. You can’t store months of data locally.

Scale: You might have thousands of devices. Monitoring each one individually doesn’t scale. You need fleet-level visibility.

Cost: Sending raw sensor data to the cloud is expensive. Cellular data plans cost money. You need to send only what matters.

All of this makes drift detection harder. But it’s not impossible. You just need the right approach.

What to Monitor from AIoT Devices

You don’t need to send everything. You need enough signals to detect when something’s wrong.

Input-Side Signals

Track simple statistics about sensor features. These are cheap to compute and small to transmit.

Rolling statistics:

• Mean and variance of each feature over a time window
• Min and max values
• Percentiles (25th, 50th, 75th, 95th)

Feature distributions:

• Simple histograms (10-20 bins per feature)
• Quantile summaries
• Count of values outside expected ranges

Example: For a vibration sensor, send the mean RMS value, peak value, and kurtosis every hour. That’s three numbers. If the mean shifts from 2.5 to 3.2, you know something changed.

Model-Side Signals

Track what the model is doing, not just what it sees.

Confidence scores over time:

• Mean confidence per class
• Distribution of confidence scores
• Count of low-confidence predictions

Prediction distribution:

• Share of predictions per class
• Count of each class per day
• Changes in class distribution

Uncertainty metrics:

• Number of predictions below a confidence threshold
• Entropy of predictions (how “confused” the model is)
• Count of “uncertain” outputs

Example: If your anomaly detector normally has 95% confidence on normal samples, but suddenly 30% of predictions are below 80% confidence, something’s wrong. Either the data changed, or the model is broken.

Business/Process Signals

These connect model behavior to real outcomes.

Ground truth events:

• Confirmed failures (when you know the model was right or wrong)
• Manual labels from operators
• Maintenance records

Operator overrides:

• When operators ignore model predictions
• When operators take action the model didn’t suggest
• When operators report false alarms

Business metrics:

• False alarm rate
• Time between true failures missed
• Cost of missed detections

Example: If operators start ignoring 40% of your alerts, either the model is too sensitive, or it’s detecting the wrong things. Either way, you need to know.

You Don’t Need a Heavy Framework

You don’t need complex drift detection on device. You need simple stats. Compute them on device. Send them periodically. Do the heavy analysis in the cloud.

On device: compute mean, variance, min, max. Send every hour.

In cloud: compare recent stats to historical baseline. Run statistical tests. Detect drift.

This keeps device code simple. It keeps telemetry small. It keeps costs low.

Designing Telemetry for Constrained AIoT Devices

You have limited bandwidth. You have limited battery. You need to send enough data to detect drift, but not so much that you break the bank.

Patterns for Low-Bandwidth Telemetry

Periodic snapshots: Send a summary every N minutes or hours. Include feature stats, model stats, and recent predictions. This gives you regular checkpoints.

Event-triggered uploads: Send immediately when something interesting happens. High confidence anomaly. Low confidence prediction. Sensor reading outside normal range. This gives you real-time alerts.

Hybrid approach: Send periodic snapshots for routine monitoring. Send event-triggered uploads for anomalies. This balances cost and responsiveness.

Compressing Histograms and Summaries

Don’t send raw histograms. Send compressed versions.

Quantile summaries: Instead of 100 histogram bins, send 5-10 quantiles. That’s enough to reconstruct the distribution shape.

Delta encoding: Send changes from the last snapshot, not absolute values. If the mean shifted by 0.3, send +0.3, not the new mean.

Sparse encoding: Only send features that changed significantly. If 20 features are stable and 2 changed, only send the 2.

Example: Instead of sending 1000 raw sensor readings, send: {"mean": 2.5, "std": 0.3, "p25": 2.2, "p50": 2.5, "p75": 2.8, "p95": 3.1}. That’s 6 numbers instead of 1000.

Using Existing Protocols

MQTT topic design: Use a clear topic structure. fleet/{site_id}/{device_id}/ml-metrics for model metrics. fleet/{site_id}/{device_id}/sensor-stats for sensor statistics. fleet/{site_id}/{device_id}/alerts for events.

This makes it easy to subscribe to what you need. You can monitor one device, one site, or the whole fleet.

Backoff and retry: When offline, queue telemetry locally. When back online, send with exponential backoff. Don’t spam the network when connectivity is spotty.

Batching: Group multiple metrics into one message. Instead of 10 separate MQTT publishes, send one JSON object with 10 metrics.

Privacy and Cost Considerations

Anonymize device IDs: If telemetry might leak, use hashed device IDs. The cloud knows the mapping. External observers don’t.

Throttle telemetry: Set rate limits. Max one message per minute. Max 100KB per day. This prevents runaway costs.

Compress payloads: Use gzip or similar. JSON compresses well. You can cut payload size by 70-80%.

Example: A 1KB JSON payload compresses to ~300 bytes. Over cellular, that’s real money saved.

Cloud-Side Drift and Quality Checks

Once telemetry arrives in the cloud, you need to detect drift. This is where the heavy lifting happens.

Simple Cloud Pipeline

Ingest: MQTT messages arrive. Parse JSON. Extract metrics. Store in time-series database (InfluxDB, TimescaleDB, or similar).

Store: Each device has a time series. Each metric is a series. Query by device, by metric, by time range.

Analyze: Run drift checks periodically. Compare recent window to reference window. Compute drift scores. Flag anomalies.

Alert: When drift is detected, send alerts. Trigger retraining. Update dashboards.

Straightforward Drift Detection Techniques

You don’t need fancy ML for drift detection. Simple statistical tests work.

Population comparison: Compare recent data to historical baseline. Use Kolmogorov-Smirnov test for distributions. Use Population Stability Index (PSI) for feature drift. Use t-tests for means.

Threshold-based: Set simple thresholds. “If more than 40% of values are outside the historical range, flag drift.” “If mean shifted by more than 2 standard deviations, flag drift.”

Trend detection: Look for gradual shifts. If mean increases by 0.1 every week for a month, that’s drift. Linear regression on rolling windows catches this.

Example: For vibration RMS values, compare last 7 days to previous 30 days. If KS test p-value < 0.05, distributions differ. That’s drift.

Connecting Drift to Business Impact

Drift detection is useless if it doesn’t connect to outcomes.

False negatives in predictive maintenance: If model confidence drops and you start missing failures, that’s business impact. Track missed failures. Correlate with drift scores.

False alarms in smart buildings: If model becomes too sensitive and operators ignore alerts, that’s business impact. Track alarm fatigue. Correlate with prediction distribution changes.

Cost of inaction: Calculate what drift costs. One missed pump failure = 8 hours downtime = $50K lost revenue. If drift increases missed failure rate by 20%, that’s $10K per month per device.

Make drift visible. Make it actionable. Make it matter.

The Model Lifecycle for AIoT Fleets

Managing models for edge devices is different from cloud models. You have versioning. You have rollouts. You have rollbacks. You have fleets.

Stages of the Lifecycle

Initial training and evaluation: Train model on historical data. Evaluate on held-out test set. Validate on similar devices. This is the baseline.

Pilot deployment: Deploy to one device or one site. Monitor closely. Compare to baseline. If metrics degrade, roll back immediately.

Fleet rollout: If pilot succeeds, roll out to more devices. Use canary or ring-based rollout. Monitor each ring before proceeding.

Periodic retraining: When drift is detected, retrain on recent data. Include data from the field. Validate on current distributions. Deploy new version.

Retirement: When model is superseded, mark old version as deprecated. Stop deploying to new devices. Eventually remove from active fleet.

Model Registry for AIoT

You need a registry that tracks versions, metadata, and deployment status.

Versioning: Each model version has:

• Unique version ID (semantic versioning: 1.2.3)
• Training data slice (which data, what time range)
• Performance metrics (accuracy, F1, etc.)
• Thresholds (confidence thresholds, anomaly thresholds)
• Device compatibility (which devices can run this version)

Deployment tracking: Track which version is live on which devices:

• Device ID → model version mapping
• Rollout ring assignment
• Deployment timestamp
• Rollback capability (can this device roll back?)

Metadata: Store everything you need to reproduce or debug:

• Training code version
• Hyperparameters
• Feature engineering steps
• Data preprocessing pipeline
• Model file hash

Example registry entry:

model_id: vibration_anomaly_v1
version: 1.2.3
trained_on: 2025-01-01 to 2025-06-30
devices: pump_fleet_a
performance:
  accuracy: 0.94
  false_positive_rate: 0.02
  false_negative_rate: 0.01
deployment:
  ring_0: 5 devices (pilot)
  ring_1: 50 devices (canary)
  ring_2: 500 devices (full fleet)

Safe OTA Rollout Patterns for Models

Rolling out model updates over-the-air is risky. A bad model can break thousands of devices. You need safe rollout patterns.

Canary Rollout

Start small. Deploy to a few devices. Monitor closely. If metrics look good, expand.

Choose canary devices:

• Pick devices that are representative of the fleet
• Pick devices you can monitor easily
• Pick devices in non-critical locations
• Pick devices with good connectivity

Compare canary to control:

• Keep some devices on old version (control group)
• Compare metrics: canary vs control
• Look for regressions: higher false alarm rate, lower confidence, more errors

Decision criteria:

• If canary metrics are better or equal: proceed to next ring
• If canary metrics are worse: roll back, investigate, fix
• If canary metrics are unclear: wait longer, gather more data

Example: Deploy v1.2.3 to 5 devices. Compare false alarm rate: canary 2%, control 2.5%. Confidence scores: canary 94%, control 93%. Proceed to next ring.

Ring-Based Rollout

Divide fleet into rings. Deploy to ring 0 first (lab/staging). Then ring 1 (friendly customers). Then ring 2 (full fleet).

Ring 0: Lab/Staging

• Test environment
• Synthetic data
• Can break without impact
• Use for initial validation

Ring 1: Friendly Customers / One Site

• Real devices, real data
• Customers who can tolerate issues
• One factory or one region
• Monitor for 1-2 weeks

Ring 2: Full Fleet

• All devices
• Only after ring 1 succeeds
• Gradual rollout (10% → 50% → 100%)
• Monitor for regressions

Ring assignment:

• Devices can be in multiple rings (for A/B testing)
• Rings can overlap (ring 1 might be subset of ring 2)
• Rings can be geographic, by device type, or by customer

Guardrails

Set up automatic safety checks. If metrics degrade, roll back automatically.

Automatic rollback conditions:

• False alarm rate increases by >50%
• Confidence scores drop by >10%
• Error rate increases by >20%
• Operator override rate increases by >30%

Safe mode: If model health is bad, fall back to rules-based logic. Instead of model predictions, use simple thresholds. “If vibration > X, alert.” This keeps the system working even if the model breaks.

Rollback procedure:

• Keep previous model version on device
• On rollback trigger, switch to previous version
• Log rollback event
• Alert operators
• Investigate root cause

Example: New model deployed. False alarm rate jumps from 2% to 5%. Automatic rollback triggers. Devices switch back to previous version. Operators notified. Team investigates why new model is too sensitive.

End-to-End Example: Vibration Anomaly Detection

Let’s walk through a complete example. A pump with an accelerometer. An anomaly detection model. Full lifecycle from training to drift handling.

Scenario

Device: Industrial pump with 3-axis accelerometer Model: Tiny autoencoder for anomaly detection Goal: Detect pump failures before they cause downtime Constraint: Device has limited CPU, memory, and battery

Feature Extraction on Device

The device reads accelerometer data. It computes features over a sliding window.

Windowing:

• Read 1024 samples at 1kHz (1 second of data)
• Slide window by 256 samples (75% overlap)
• Compute features for each window

Features:

• RMS (root mean square) of each axis
• Peak value of each axis
• Kurtosis (tailedness) of each axis
• Frequency domain features (FFT, dominant frequency)

Code structure:

# Pseudocode for device
window = read_accelerometer(1024 samples)
features = {
    'rms_x': compute_rms(window.x),
    'rms_y': compute_rms(window.y),
    'rms_z': compute_rms(window.z),
    'peak_x': max(abs(window.x)),
    'peak_y': max(abs(window.y)),
    'peak_z': max(abs(window.z)),
    'kurtosis_x': compute_kurtosis(window.x),
    # ... more features
}

Model Training in the Cloud

Train a simple autoencoder. It learns to reconstruct normal vibration patterns. When patterns are abnormal, reconstruction error is high. That’s the anomaly score.

Model architecture:

• Input: 9 features (3 axes × 3 feature types)
• Encoder: 9 → 6 → 3 (compression)
• Decoder: 3 → 6 → 9 (reconstruction)
• Loss: Mean squared error between input and output

Training:

• Use normal vibration data (no failures)
• Train for 100 epochs
• Validate on held-out normal data
• Set threshold: 95th percentile of reconstruction error on validation set

Export:

• Convert to TensorFlow Lite Micro
• Quantize to int8 (reduce model size)
• Test on device simulator
• Package with metadata

Deployment

OTA update process:

1. Upload model to cloud registry
2. Assign to rollout ring (start with ring 0)
3. Devices check for updates periodically
4. Device downloads model
5. Device verifies signature
6. Device loads model
7. Device starts using new model
8. Device reports metrics

Device-side loading:

# Pseudocode
def load_model(model_version):
    # Download from cloud
    model_bytes = download_model(model_version)
    
    # Verify signature
    if not verify_signature(model_bytes):
        return None
    
    # Load into TensorFlow Lite Micro
    interpreter = tflite.Interpreter(model_bytes)
    interpreter.allocate_tensors()
    
    return interpreter

Drift Handling

Telemetry: Device sends feature statistics every hour:

{
  "device_id": "pump-001",
  "timestamp": "2025-11-26T10:00:00Z",
  "features": {
    "rms_x": {"mean": 2.5, "std": 0.3, "min": 2.0, "max": 3.1},
    "rms_y": {"mean": 1.8, "std": 0.2, "min": 1.5, "max": 2.2},
    "rms_z": {"mean": 3.2, "std": 0.4, "min": 2.5, "max": 4.0}
  },
  "model": {
    "version": "1.2.3",
    "confidence_scores": {"mean": 0.94, "std": 0.05},
    "anomaly_rate": 0.02
  }
}

Cloud-side drift detection:

# Compare recent week to previous month
recent_stats = get_stats(device_id, last_7_days)
baseline_stats = get_stats(device_id, previous_30_days)

# KS test for distribution shift
for feature in ['rms_x', 'rms_y', 'rms_z']:
    ks_stat, p_value = ks_test(
        recent_stats[feature]['distribution'],
        baseline_stats[feature]['distribution']
    )
    if p_value < 0.05:
        flag_drift(device_id, feature, 'distribution_shift')

# Threshold check for mean shift
for feature in ['rms_x', 'rms_y', 'rms_z']:
    mean_shift = abs(
        recent_stats[feature]['mean'] - 
        baseline_stats[feature]['mean']
    )
    if mean_shift > 2 * baseline_stats[feature]['std']:
        flag_drift(device_id, feature, 'mean_shift')

Retraining trigger: When drift is detected:

1. Collect recent data from affected devices
2. Retrain model on recent data + historical data
3. Validate on current distribution
4. Deploy new version via canary rollout

Metrics, Dashboards, and SLOs for AIoT Models

You need to measure model health. You need to set targets. You need to know when things are broken.

Realistic Metrics

False alarm rate: Number of false alarms per device per week. Target: < 5 per week. If it’s higher, model is too sensitive.

Time between true failures missed: Average time between when model should have caught a failure but didn’t. Target: > 90 days. If it’s lower, model is missing real issues.

Confidence score distribution: Mean and std of confidence scores. Target: mean > 90%, std < 10%. If confidence drops, model is uncertain.

Telemetry volume: Bytes sent per device per day. Target: < 100KB per day. If it’s higher, you’re sending too much data.

Model update success rate: Percentage of devices that successfully update. Target: > 95%. If it’s lower, OTA process has issues.

Dashboards

Fleet overview:

• Total devices
• Devices per model version
• Devices per rollout ring
• Overall false alarm rate
• Overall confidence scores

Drift detection:

• Devices with detected drift
• Drift scores over time
• Features with highest drift
• Correlation with business metrics

Model performance:

• False alarm rate by version
• Confidence scores by version
• Anomaly detection rate by version
• A/B test results (canary vs control)

Device health:

• Devices online/offline
• Telemetry latency
• Update success/failure
• Error rates

Simple SLOs

Set service level objectives. These are your targets.

Model accuracy SLO:

• 95% of predictions have confidence > 80%
• 99% of true failures are detected within 24 hours
• False alarm rate < 5 per device per week

System reliability SLO:

• 99.9% of devices online
• 95% of OTA updates succeed
• Telemetry latency < 5 minutes (p95)

Cost SLO:

• Telemetry < 100KB per device per day
• Model update bandwidth < 1MB per device per update

When SLOs are violated, that’s when you act. That’s when you investigate. That’s when you fix.

Practical Checklist and Templates

Here’s a checklist to get started. Use it as a template. Adapt it to your system.

Pre-Deployment Checklist

• Do you have a reference data window? (30-90 days of “normal” data)
• Which stats do you send from device? (mean, variance, min, max, percentiles)
• What are your drift thresholds? (KS test p < 0.05, mean shift > 2σ)
• What’s your OTA + rollback story? (canary → ring 1 → ring 2, automatic rollback)
• Do you have a model registry? (versions, metadata, deployment tracking)
• Do you have dashboards? (fleet overview, drift detection, model performance)
• Do you have SLOs? (false alarm rate, confidence scores, update success rate)
• Do you have alerting? (drift detected, SLO violation, update failure)

Telemetry Template

{
  "device_id": "device-001",
  "timestamp": "2025-11-26T10:00:00Z",
  "model_version": "1.2.3",
  "features": {
    "feature_1": {
      "mean": 2.5,
      "std": 0.3,
      "min": 2.0,
      "max": 3.1,
      "p25": 2.2,
      "p50": 2.5,
      "p75": 2.8,
      "p95": 3.0
    }
  },
  "model_metrics": {
    "mean_confidence": 0.94,
    "std_confidence": 0.05,
    "predictions_per_class": {
      "normal": 1000,
      "anomaly": 20
    },
    "low_confidence_count": 5
  }
}

Drift Detection Template

def detect_drift(device_id, feature_name, recent_window_days=7, baseline_window_days=30):
    recent = get_feature_stats(device_id, feature_name, recent_window_days)
    baseline = get_feature_stats(device_id, feature_name, baseline_window_days)
    
    # KS test
    ks_stat, p_value = ks_test(recent['distribution'], baseline['distribution'])
    
    # Mean shift
    mean_shift = abs(recent['mean'] - baseline['mean'])
    mean_shift_sigma = mean_shift / baseline['std']
    
    # Variance shift
    variance_ratio = recent['variance'] / baseline['variance']
    
    drift_detected = (
        p_value < 0.05 or  # Distribution changed
        mean_shift_sigma > 2.0 or  # Mean shifted by >2σ
        variance_ratio > 1.5 or variance_ratio < 0.67  # Variance changed significantly
    )
    
    return {
        'drift_detected': drift_detected,
        'ks_p_value': p_value,
        'mean_shift_sigma': mean_shift_sigma,
        'variance_ratio': variance_ratio
    }

Deployment Config Template

model_id: vibration_anomaly_v1
version: 1.2.3
trained_on: 2025-01-01 to 2025-06-30
rollout:
  ring_0:
    devices: ["pump-test-001", "pump-test-002"]
    start_date: 2025-11-26
    status: active
  ring_1:
    devices: ["pump-factory-a-*"]  # All devices in factory A
    start_date: 2025-12-01
    status: pending
  ring_2:
    devices: ["pump-*"]  # All pumps
    start_date: 2025-12-15
    status: pending
rollback_conditions:
  - false_alarm_rate_increase: 0.5  # 50% increase
  - confidence_drop: 0.1  # 10% drop
  - error_rate_increase: 0.2  # 20% increase

Conclusion

Models on edge devices drift. It’s not a question of if, but when. Sensor patterns change. Environments shift. Hardware ages. Models become blind.

You can’t prevent drift. But you can detect it. You can retrain. You can roll out updates safely. You can roll back when things go wrong.

Start simple. Send feature statistics from devices. Compare recent data to baseline in the cloud. Set up alerts. When drift is detected, retrain. Deploy via canary rollout. Monitor. Roll back if needed.

You don’t need a perfect system on day one. Start with basic telemetry. Add drift detection. Add retraining. Add safe rollouts. Iterate.

The alternative is models that break silently. Devices that miss failures. Systems that degrade over time. That’s not acceptable.

Monitor. Update. Roll back. That’s how you keep edge models useful after you ship them.