10 ML Fundamentals and Validation Interview Questions and Answers

Short answer: A model's expected error decomposes into bias² + variance + irreducible noise. Bias is the systematic error of a too-simple model (underfitting), variance is sensitivity to the noise of a particular sample (overfitting). Reducing one usually raises the other; the job is to find the balance.

In depth:

1. Decomposition — E[(y − ŷ)²] = bias² + variance + σ²; the noise σ² cannot be fixed by any model.
2. High bias — the model is simpler than the true relationship: bad on both train and validation. Example: linear regression on nonlinear data.
3. High variance — the model fits noise: near-perfect train, validation collapses. Example: an unconstrained deep tree.
4. Levers — model complexity, regularization, ensembles (bagging cuts variance). More data reduces variance but not bias.
5. Diagnosis — the train/validation gap: a large gap → variance; both errors high → bias.

error │ bias² ↓             ↑ variance
      │  ─._            _.─'
      │     '─.______.─'  ← sweet spot
      └───────────────────► model complexity

⚠️ Common mistake: reciting the decomposition formula without connecting it to over- and underfitting. The interviewer wants diagnosis (learning curves, train/val gap) and levers, not a memorized definition.

Short answer: Overfitting means the model learned the noise of the training set instead of the pattern: low error on train, high error on new data. You detect it via the train/validation gap and fix it with regularization, a simpler model, early stopping, and more data.

In depth:

1. Detection — compare train vs validation error; learning curves: train error keeps falling while validation error stalls or rises → the model started memorizing noise.
2. Treatment — reduce capacity or add signal, don't just chase the metric.

⚠️ Common mistake: saying "cross-validation prevents overfitting." CV only detects it — it gives an honest quality estimate. Prevention comes from regularization, simpler models, and data.

Short answer: L1 (lasso) penalizes Σ|w| and produces exact zeros — a sparse model with built-in feature selection. L2 (ridge) penalizes Σw² and smoothly shrinks all weights without zeroing them; it behaves more stably with correlated features. L1's zeros follow from the geometry of the penalty.

In depth:

1. Geometry — the L1 level set is a diamond with corners on the coordinate axes; the optimum lands in a corner with high probability, where some weights are exactly 0. The L2 ball is smooth — the tangent point almost never lies on an axis.
2. Subgradient — the derivative of |w| near zero is constant (±λ): the penalty pushes with the same force right at zero, so driving a weight all the way to zero pays off. For L2 the penalty gradient 2λw vanishes as w → 0 — the weight only approaches zero asymptotically.
3. Bayesian view — L1 = Laplace prior (sharp peak at zero), L2 = Gaussian prior.
4. Correlated features — L1 arbitrarily picks one from a group, L2 splits the weight among them; when you need both, use elastic net.

L1: |w₁|+|w₂| ≤ t          L2: w₁²+w₂² ≤ t
        ◆  corners on axes       ●  smooth circle
optimum often in a corner   tangent off-axis
→ w₂ = 0 (exact zero)       → no zeros, shrinkage

⚠️ Common mistake: saying "L1 selects features" without explaining the mechanism. The differentiator is exactly "why zeros": the interviewer expects the geometry or the subgradient argument.

Short answer: Split the data into k folds; train k times on k−1 folds, validate on the held-out one, and average the metric. The scheme is only valid for independent observations: time series, grouped data, and heavy class imbalance require special splitters.

In depth:

1. Mechanics — every point lands in validation exactly once; you get a quality estimate plus its spread across folds, not a single number.
2. Time series — shuffling puts the future into train: the model "sees" what doesn't exist yet in reality, so the estimate is inflated. Use forward-chaining: train strictly before test.
3. Groups — the same user (patient, device) ends up in both train and validation: the model recognizes the user, not the pattern. A group must go entirely into one fold.
4. Class imbalance — in small folds the rare class can vanish; stratification preserves class proportions in every fold.

from sklearn.model_selection import (KFold, TimeSeriesSplit,
                                     GroupKFold, StratifiedKFold)
KFold(5, shuffle=True)   # i.i.d. data — the baseline
TimeSeriesSplit(5)       # time: train strictly before test
GroupKFold(5)            # a user/patient never spans folds
StratifiedKFold(5)       # imbalance: class ratios preserved

⚠️ Common mistake: explaining the mechanics but not knowing the caveats. The question almost always continues with "and for time series?" — without forward-chaining the answer doesn't count.

Short answer: The validation set exists for hyperparameter tuning and model selection. If you tune on the test set, you overfit the test set: every decision fits the model to that particular sample, and the reported metric becomes optimistic. The test set is touched exactly once — at the very end.

In depth:

1. Three roles — train fits the weights; validation compares models and hyperparameters; test gives an unbiased estimate of the final model.
2. Why the test set "burns" — picking the best of a hundred configurations by test metric also picks the one that got lucky on exactly those points. The test metric stops predicting performance on new data.
3. Validation "burns" too — but that's its job: it gets spent on decisions, while the test set stays clean for reporting.
4. Little data — use cross-validation on the training set instead of a fixed validation split; the test set is still kept separate.

train ──► fit the weights
val   ──► model and hyperparameter selection (many times)
test  ──► final evaluation (exactly once)

⚠️ Common mistake: treating validation and test as interchangeable — "I do have a holdout." If decisions were made against it, it's a validation set, and an honest estimate needs an untouched test set.

Short answer: Grid search enumerates a grid and explodes combinatorially. Random search with the same budget usually wins in high dimensions: only a few hyperparameters actually matter, and random sampling tries more values of those (Bergstra & Bengio). Bayesian optimization (Optuna) models "parameters → score" and spends trials on promising regions. Tune only on validation/CV, never on the test set.

In depth:

1. Why random beats grid — if 2 of 5 hyperparameters matter, a 3×3×…×3 grid tries only 3 unique values of each important one, while 243 random points try 243 values.
2. When grid is fine — 1–2 parameters, a cheap model, and you want a reproducible quality map.
3. Bayesian search — surrogate model + acquisition function: balances exploring and exploiting the best zones; pays off when training is expensive. Plus pruning of obviously weak configurations.

⚠️ Common mistake: tuning hyperparameters against the test set. That's leakage: the test set took part in model selection, so the reported metric is inflated.

Short answer: Four prime suspects: leakage inside CV (preprocessing fit on all data, duplicates across folds), distribution shift over time, overfitting through the analyst's many CV-driven decisions, and group leakage. Each hypothesis has its own concrete check.

In depth:

1. Order of operations — cheap checks first (duplicates, PSI), then rebuild the pipeline and re-validate honestly.
2. Red flag — a CV metric that is "too good" relative to the baseline and common sense: more often leakage than a genius model.

⚠️ Common mistake: answering with the single word "overfitting." The interviewer wants mechanisms and checks: how exactly the CV estimate could be inflated and how to confirm it.

Short answer: As the number of features grows, the volume of the space grows exponentially: data becomes sparse and pairwise distances concentrate — the "nearest" neighbor is barely closer than the farthest one. Distance-based methods suffer most: kNN, k-means, kernel methods.

In depth:

1. Sparsity — to keep the same coverage density, the amount of data must grow exponentially with dimension; in practice you never have that much.
2. Distance concentration — in high dimensions the ratio (max − min)/min of distances tends to zero: "nearest" loses meaning, and kNN with it.
3. Who suffers and who holds up:

4. Mitigations — feature selection, PCA/embeddings, regularization, more data.

⚠️ Common mistake: dropping the buzzword without a consequence. The interviewer wants specifics: what exactly breaks (distances, density) and what you do about it.

Short answer: Generative models learn the joint distribution P(x, y) and obtain P(y|x) via Bayes' rule; discriminative models learn P(y|x) or the decision boundary directly. Naive Bayes is generative, logistic regression is discriminative; with enough data, the discriminative one usually wins at pure classification.

In depth:

1. What generative buys you — the model "knows" how the data itself is structured: it can generate samples, handle missing features (by integrating them out), and converge fast on small samples if its assumptions hold.
2. What discriminative buys you — no assumptions about the distribution of x: all capacity goes into the class boundary, so asymptotic accuracy is usually higher as data grows (the classic reference — Ng & Jordan, 2001).
3. Naive Bayes — assumes conditional independence of features; the assumption is almost always false, yet the model is cheap and surprisingly resilient (spam filters).

⚠️ Common mistake: misassigning which is which — calling logistic regression generative because "it's probabilistic." Probabilistic ≠ generative: what matters is what is modeled — P(x, y) or P(y|x).

Short answer: Parametric models have a fixed number of parameters, independent of dataset size — the functional form is chosen in advance (linear and logistic regression). Non-parametric models grow their capacity with the data (kNN, trees): fewer assumptions, but a bigger appetite for data.

In depth:

1. Parametric — a strong assumption about the form (e.g. "the logit is linear in features"): fast to train, needs little data, robust to noise — but if the form is guessed wrong, it hits a ceiling (high bias).
2. Non-parametric — the form "grows" out of the data: kNN stores the whole training set, a tree keeps adding splits. Flexibility gives low bias but raises variance and the data requirement.
3. Extrapolation — a parametric model extends its formula beyond the training data; kNN and trees output a constant outside the data region — they cannot predict "outward."

⚠️ Common mistake: "non-parametric = no parameters." There are parameters — their count just isn't fixed and grows with the sample: kNN stores every point, a tree stores every split.