8 ML System Design Interview Questions and Answers

Short answer: Don't start with a model. Ask clarifying questions first: what ACTION will the business take with the prediction — it sets the prediction horizon and the label window. Then a non-ML baseline, target definition, data, an offline metric tied to money, and experiment design.

In depth:

1. The action — who does what with the score (call, discount, push)? The action defines the horizon and the cost of errors.
2. Target definition — what counts as "churn": no purchases for 60 days? unsubscribing? Fix the observation window explicitly.
3. Non-ML baseline — the rule "inactive for 30 days" may capture 80% of the value; the model must beat it.
4. Data and features — only what is known at prediction time (as-of-time), otherwise leakage.
5. Metric — an offline metric (PR-AUC, precision@k) plus its link to money: cost of a retention offer vs a false positive.
6. Validation and experiment — time-based split, then an A/B test on the actual action.

business ask → action → target & horizon → baseline (rules)
    → as-of-time data → offline metric ↔ money → time split → A/B

⚠️ Common mistake: jumping to "let's train XGBoost" before the target and the action are defined — the Yandex ML section fails candidates exactly at this step.

Short answer: A two-stage cascade: candidate retrieval (millions → hundreds) with light models, then heavy ranking of the candidates with boosting or a neural net, business rules on top. The main pitfalls: cold start, position bias in implicit feedback, and the offline/online metric gap.

In depth:

1. Candidate retrieval — collaborative filtering (ALS), two-tower embeddings + an ANN index (HNSW/FAISS): cheap, high recall.
2. Ranking — boosting/NN on user, item and context features; the target is click/purchase corrected for position bias.
3. Business rules — diversity, freshness, filters (already purchased), category quotas.
4. Cold start — content features for new items, popularity priors for new users.
5. Metrics — offline recall@k/NDCG on logs ≠ online CTR/GMV: logs are biased by the previous model (feedback loop); the A/B test has the final word.

catalog (10⁶)
   │  candidate retrieval: ALS / two-tower + ANN
   ▼
candidates (10²–10³)
   │  ranking: boosting/NN, user × item × context features
   ▼
top-N → business rules (diversity, freshness) → feed

⚠️ Common mistake: a one-stage answer — "the model ranks the whole catalog" — with no word on cold start; it instantly signals no production experience.

Short answer: Fraud detection means extreme class imbalance, a hard latency budget, DELAYED labels, and an adversary that adapts. The design: a feature store with precomputed aggregates + light online features, a model + a rules layer, precision@k sized to the review team's capacity, frequent retraining, and human-in-the-loop.

In depth:

1. Human-in-the-loop — score → auto-block / analyst queue / pass; analyst labels feed back into training.

⚠️ Common mistake: designing a "plain classifier" while ignoring label delay and drift — in fraud detection they are the core of the problem, not a detail.

Short answer: The constraints around the model: the regulator demands interpretability (scorecards, WoE, monotonicity), the metric language is Gini, the score is calibrated to PD (probability of default), population stability matters more than the last percent of accuracy, and you never see labels for rejected applicants (reject inference).

In depth:

1. Interpretability — logistic regression on WoE features or boosting with monotonic constraints; every feature is justified (IV), and a rejection reason must be explainable.
2. Metric language — Gini (= 2·AUC − 1): bank teams say "Gini 55", not "AUC 0.775".
3. Calibration to PD — the default probability drives the rate, the limit, and reserves: calibration matters more than ranking.
4. Stability — PSI tracks population shift; an unstable feature gets dropped even if it is strong.
5. Reject inference — training only on approved applicants = selection bias; rejected ones are labeled with surrogates or modeled.
6. Label lag — default is observed 12+ months later: vintage-based validation.

⚠️ Common mistake: "I'll take CatBoost and maximize AUC" — in a bank a model without calibration, PSI, and explainability will not pass validation.

Short answer: A response model predicts P(buy | treated); uplift models the CHANGE in probability caused by the treatment itself (CATE): P(buy | promo) − P(buy | no promo). You should target only the persuadables — the people the promo actually flips.

In depth:

1. Approaches — S-learner (treatment as a feature), T-learner (two models), X-learner, uplift trees with a split criterion on the response difference.
2. Data — you need a randomized experiment: without random treatment assignment the CATE is not identifiable.
3. Validation — uplift@k, the Qini curve: plain AUC/logloss don't work because an individual's true uplift is unobservable.

⚠️ Common mistake: targeting people with high P(buy) — those are sure things: the campaign's conversion looks great, the incremental effect is zero. In the CIS (retail, telecom, banks) this question is popular precisely because of that trap.

Short answer: Three ways to turn ranking into a learnable task: pointwise predicts each document's relevance independently, pairwise learns from preference pairs ("A is more relevant than B"), listwise optimizes the whole-list metric (NDCG) directly.

In depth:

1. LambdaRank — the bridge between pairwise and listwise — the pair gradient is scaled by |ΔNDCG| from swapping them: formally pairwise, effectively optimizing the list metric.
2. In practice — LambdaMART (boosting + LambdaRank) is the workhorse of search; CatBoost ships YetiRank/PairLogit ranking modes.
3. Tie to the metric — NDCG discounts position logarithmically: mistakes at the top cost more, which is why pointwise MSE correlates poorly with result quality.

⚠️ Common mistake: "I'll fit a regression on clicks and sort by score" with no word on order, position, or NDCG — to a search team that is an intern-level answer.

Short answer: The classic offline-online gap. Hypotheses in checking order: leakage or a wrong validation scheme, logged-data biases (position/selection bias), the model is not the product's bottleneck, the effect is below the experiment's power, latency degradation ate the gain, feedback loop.

In depth:

⚠️ Common mistake: immediately declaring "the A/B test is broken" or "the offline metric is useless" — the interviewer expects a structured list of hypotheses with cheap checks, from likely to exotic.

Short answer: When a rule or heuristic captures the value more cheaply: there are no labels or the feedback is too slow, the cost of errors demands explainability and guarantees, data is scarce, and the model's maintenance cost (drift, retraining, monitoring) exceeds its gain over the baseline.

In depth:

Checklist "does this need ML":
□ Is there a non-ML baseline and how much value does it capture?
□ Are there labels? How fast does feedback arrive?
□ Can the task tolerate probabilistic errors? Is explainability required (regulator)?
□ Is there enough data for a signal beyond a rule?
□ Who maintains it: drift, retraining, monitoring, on-call?
□ Does the gain over the rule pay for all of the above?

1. Baseline first — always name the heuristic before the model: "sort by popularity", "the 30-day rule". Often it is enough.
2. ML is a commitment — a process, not an artifact: data drifts, pipelines break, models rot without retraining.
3. What the interviewer screens for — maturity: a candidate with an ML-shaped hammer for every problem is a red flag; this question filters for seniority.

⚠️ Common mistake: reflexively proposing a model to a question that began with "the business wants…" — rule first, then ML if the rule falls short.