10 MLOps and Production Interview Questions and Answers

Short answer: Data drift (covariate shift) — the input distribution P(x) changes: say, a new user segment shows up. Concept drift — the relationship itself P(y|x) changes: fraud tactics evolve, tastes shift. The first is visible immediately in the features; the second only shows up in quality on fresh labels.

In depth:

1. PSI (Population Stability Index) — rule of thumb: PSI < 0.1 stable, 0.1–0.2 worth a look, > 0.2 alert.
2. Score monitoring — a shift in the prediction distribution often shows up earlier than shifts in individual features.
3. Delayed quality evaluation — concept drift is invisible without labels; track metrics on matured cohorts.

⚠️ Common mistake: treating any drift as a reason to retrain. Drift in a zero-importance feature doesn't touch quality — assess the impact first, then act.

Short answer: While there are no labels, monitor proxies: input feature distributions, shifts in the score distribution, prediction volume per segment. Measure quality with a delay — on cohorts whose labels have matured. Alert on distributions first, on quality later.

In depth:

1. Inputs — PSI/KS on key features: a broken data source is visible within hours, not months.
2. Outputs — score distribution and approve/block rates per segment; a sharp shift in the mean score is an early signal.
3. Delayed evaluation — cohort approach: quality for January's loans is computed in April, once defaults have matured; watch the cohort-over-cohort trend.
4. Leading indicators — metrics correlated with the target: first payment default (FPD) as a proxy for default, chargeback claims as a proxy for fraud.

day 0         days 1–90            day 90+
predict ──►  proxy monitoring ──►  labels matured
             (features, scores,     └─► cohort
              volumes)                  quality

⚠️ Common mistake: "we'll just track accuracy" — there are no labels yet, there is nothing to track. The interviewer wants a plan precisely for the blind period.

Short answer: Two approaches: on a schedule (weekly/monthly) or on a trigger (drift alert, quality degradation). Either way the new version is a challenger: validated on fresh data and promoted to production only through an eval gate, never automatically.

In depth:

1. On a schedule — simple and predictable; fits when data drifts smoothly. Risk: retraining for nothing or, conversely, too late.
2. On a trigger — react to a PSI alert or a metric drop; more infrastructure, but you retrain exactly when it's needed.
3. Champion/challenger — the current model (champion) stays; the candidate is checked on a fresh-data holdout and/or in shadow mode; promotion only if it is consistently better.

Automated retraining pitfalls:

⚠️ Common mistake: "we retrain daily, so everything is fresh" — without a validation gate, daily retraining just ships bugs to production faster.

Short answer: A mismatch between the features the model was trained on and the features it receives in production: offline they were computed with batch SQL, online they were rewritten in service code — and the logic diverged. The model silently degrades without a single error in the logs. Interviewers read knowing this problem as a marker of real production experience.

In depth:

1. Where it comes from — two implementations of one feature (SQL for training, Java/Go for inference), different data sources, different null handling, "as of now" vs "as of event time" aggregates.
2. Feature store — the feature is defined once and served to both training and online inference; the main argument for adopting one.
3. Log-and-train — log the features actually served at inference time and train on those logs: training sees exactly what production saw.
4. Parity monitoring — regularly compare offline and online values of the same feature on the same entities.

              ┌── batch SQL ─────► training      ✗ two paths —
raw data ─────┤                                    logic diverges
              └── service code ──► inference

              ┌───────────────┐
raw data ─────┤ feature store ├─► training + inference   ✓ one path
              └───────────────┘

⚠️ Common mistake: checking only the schema (types match — done). Skew lives in the semantics: the same float, computed differently.

Short answer: A ladder: shadow mode (score live traffic, take no actions) → canary on a small share of traffic → A/B test on the business metric. Every rung has exit criteria and a kill switch with a rollback plan.

In depth:

1. Shadow — the model sees live traffic, responses are logged but never applied: validate latency, error rate, and the sanity of the score distribution.
2. Canary — 1–5% of traffic gets the new model's decisions; watch system and product metrics, roll back instantly on an incident.
3. A/B on the business metric — the final verdict comes from conversion/revenue/retention, not offline AUC: offline gains routinely fail to reach the product.
4. Interleaving — for ranking: blend both models' results into one list; more sensitive than a classic A/B.

offline eval ─► shadow ─► canary 1–5% ─► A/B ─► 100%
                  │            │          │
                  └────────────┴──────────┴──► rollback (kill switch)

⚠️ Common mistake: going straight to A/B without a shadow stage — you find out about latency spikes or broken features on live users.

Short answer: Four layers: inputs (feature distributions, null rates, schema and freshness), outputs (score distribution, prediction volumes per segment), system (latency, throughput, errors), and quality (metrics on arriving ground truth, per segment). Each layer gets its own alert thresholds.

In depth:

1. Segmentation is mandatory — the global metric can sit still while a key segment is on fire.
2. Feature logs — store the features served at inference: without them you can't debug an incident or retrain honestly.

⚠️ Common mistake: monitoring only "accuracy". Labels are delayed, but the pipeline breaks today — input alerts fire first.

Short answer: Data first, model last. Order: (1) did the data pipeline break — schema, nulls, a lagging source; (2) the shape of the drop — a step change (breakage/release) vs slow decay (drift); (3) input distribution shift; (4) problems with the labels themselves; (5) the world actually changed.

In depth:

1. Data pipeline — the most common cause: upstream changed the schema, a feature turned into nulls, a source is a day behind. Also the fastest to check.
2. Dating and shape — overlay the drop on the release calendar: a step change on deploy day is a breakage, not drift.
3. Input distributions — PSI per feature, look for a new traffic segment.
4. Labels — did the labeling or the target join break: sometimes a "quality drop" is a drop in measurement quality.
5. The world changed — concept drift as the last-resort hypothesis, once the boring causes are ruled out.

step change ──► look for a release or breakage that day
slow decay  ──► look for data or behavior drift

⚠️ Common mistake: "let's retrain" as the first move — if the pipeline is broken, retraining on corrupted data locks the breakage in.

Short answer: Versions of everything the result is built from: data, code, config with hyperparameters, random seeds, and the environment. Plus experiment tracking and a model registry, so every production model has a full lineage.

In depth:

1. Data — snapshots or versioning (DVC, lakeFS): "the same table" a month later is different data.
2. Code — a git commit tied to the experiment run.
3. Config — hyperparameters, feature versions, splits — in a config, not in a notebook.
4. Seeds — fix the random seed everywhere (numpy, the framework, the splits); remember GPU nondeterminism.
5. Environment — a dependency lock file or a Docker image.
6. Tracking — MLflow/W&B: parameters, metrics, and artifacts of every run.
7. Registry — the production model points to a specific run: data + code + config are recoverable.

data (v) + code (git) + config + seed + env
        └──► tracked run ──► model registry ──► prod

⚠️ Common mistake: "but I still have the notebook" — a notebook with cells re-run out of order reproduces nothing.

Short answer: This tests communication skill, not the definition. The working analogy: a student memorized last years' exam answers — aces the mock exam and fails the real one, because they learned the answers, not the subject. The model "memorized" historical data instead of the pattern.

In depth:

1. The analogy is the test — the interviewer watches whether you find an image from the listener's world: cramming vs understanding, a suit tailored strictly to one person.
2. Metrics → business language — not "AUC 0.93" but "we catch 80% of fraud and wrongly flag 1 in 200 legitimate users".
3. What it means for the product — "on historical data the model looks better than it will behave with real users, which is why we keep an honest check on held-out data".
4. Format — under a minute, not a single term; end with an action: what we do to catch it.

⚠️ Common mistake: sliding into "we'll add regularization and reduce variance" two sentences in — the jargon is back, the test is failed.

Short answer: Through a feedback loop: the model's predictions determine which data gets generated at all. The recommender shows items — users click what was shown — the clicks become the next model's training set. Anti-fraud blocks — blocked fraud never gets a label.

In depth:

1. Recommender systems — position bias and selection bias: the model trains on clicks over its own output, the filter bubble self-reinforces.
2. Fraud/scoring — reject inference: rejected applications have no outcome, so the next model's sample is biased toward the approved.
3. Mitigation — keep an exploration slice: a small share of traffic with randomization or a relaxed policy yields unbiased data.
4. Log propensities — the probability of showing/acting at decision time: enables weighting (IPS) and off-policy evaluation.

model ──► impressions/decisions ──► user behavior ──► logs
  ▲                                                     │
  └──────────────── next training ◄─────────────────────┘
     break it: exploration slice + propensity logs

⚠️ Common mistake: not noticing the loop exists at all, and celebrating gains in offline metrics the model painted for itself.