12 Classical ML and Ensembles Interview Questions and Answers

Short answer: Minimize the sum of squared errors ||y − Xw||²: set the gradient with respect to w to zero and get the normal equation XᵀXw = Xᵀy, hence w = (XᵀX)⁻¹Xᵀy. The solution breaks when XᵀX is singular — under multicollinearity or p > n.

In depth:

L(w) = ||y − Xw||² = (y − Xw)ᵀ(y − Xw)
∇w L = −2Xᵀ(y − Xw) = 0
XᵀXw = Xᵀy              ← normal equation
w* = (XᵀX)⁻¹Xᵀy

1. When the inverse doesn't exist — XᵀX is singular if features are linearly dependent (multicollinearity) or there are more features than observations (p > n): infinitely many solutions.
2. Ridge as the cure — the penalty λ||w||² gives w = (XᵀX + λI)⁻¹Xᵀy; the matrix XᵀX + λI is positive definite and invertible for any λ > 0.
3. Gradient descent — for large p the inversion costs O(p³); iterative optimization is cheaper and doesn't require invertibility at all.

⚠️ Common mistake: writing the formula but failing to say when the inversion fails. The question is almost always asked for that follow-up.

Short answer: A linear score is passed through a sigmoid: p = σ(wᵀx) — giving a class probability. We train by maximizing the Bernoulli likelihood, which is equivalent to minimizing log-loss — a convex function. MSE on top of a sigmoid is non-convex and produces vanishing gradients on confidently wrong answers.

In depth:

1. The model — p(y=1|x) = σ(wᵀx), where σ(z) = 1/(1+e⁻ᶻ); the decision boundary is still linear in feature space.
2. Loss from likelihood — the Bernoulli likelihood ∏ pʸ(1−p)¹⁻ʸ; its negative log gives log-loss −[y·log p + (1−y)·log(1−p)] — convex in w, a single minimum.
3. Why not MSE — (σ(wᵀx) − y)² is non-convex in w, and its gradient carries the factor σ′(z), which is near zero for large |wᵀx|: learning almost stops exactly on confidently wrong answers.

log-loss: ∇ = (p − y)·x         ← never vanishes
MSE:      ∇ = (p − y)·σ′(z)·x   ← σ′ ≈ 0 on confident errors

4. Bonus — the output is interpretable as an (approximately) calibrated probability, which SVMs or trees don't give you out of the box.

⚠️ Common mistake: "it's a regression, it predicts a continuous target." It's a linear classifier; the "regression" refers to regressing the logit.

Short answer: Greedily: at each node it scans features and thresholds and picks the split with the largest impurity reduction (Gini or entropy; MSE for regression). With unlimited depth the tree drives leaves to purity — i.e. it memorizes the training set, noise included.

In depth:

1. Split criterion — the gain: impurity(parent) − weighted impurity(children); Gini G = 1 − Σpₖ², entropy −Σpₖ·log pₖ — nearly interchangeable in practice.
2. Greediness — each node is optimized in isolation; the globally optimal tree is NP-hard, so a bad split near the root is never revisited.
3. Why overfitting — the model is piecewise-constant: with unlimited depth every leaf shrinks to a single sample → zero training error, high variance on test.

⚠️ Common mistake: believing the tree finds the globally "best partition" — it's greedy, and that's a fundamental limitation, not an implementation detail.

Short answer: Bagging trains independent models on bootstrap samples and averages them — that cuts VARIANCE. Boosting builds models sequentially, each one fixing the current ensemble's errors — that cuts BIAS.

In depth:

1. Why different tree depths — bagging needs diverse strong models: averaging will remove their variance. Boosting needs weak ones: it removes bias by itself, and extra per-step capacity is a straight road to overfitting.
2. Practical consequence — RF is nearly insensitive to the number of trees; in boosting, n_estimators and learning rate are the main knobs.

⚠️ Common mistake: "boosting is just bagging with weights." The key is sequential dependency: each model sees the previous models' errors, not an independent sample.

Short answer: Boosting has the higher quality ceiling on tabular data — the default in competitions and production. Random forest is the robustness pick: nearly insensitive to hyperparameters and noisy targets, trivially parallel, and hard to overfit.

In depth:

1. Why RF subsamples features (max_features) — without it, all trees would pick the same strong splits and end up correlated, and averaging correlated models barely reduces variance. Feature subsampling decorrelates the trees — that's the only reason averaging works.
2. In practice — quick baseline, noisy labels, little time → RF; squeezing the maximum out of a clean tabular dataset → CatBoost/LightGBM/XGBoost with tuning.

⚠️ Common mistake: "boosting is always better." On small noisy datasets a default-settings forest often beats an under-tuned boosting model.

Short answer: The forest barely notices: trees are independent, the prediction is an average, and losing one term shifts it by ~1/1000. Boosting breaks: every later tree was fit to residuals that included the first tree's contribution, so without it the whole sum is systematically shifted.

In depth:

RF:   ŷ = (t₁ + t₂ + … + t₁₀₀₀)/1000
      drop t₁ → shift on the order of one tree's contribution

GBM:  F = f₁ + ν·f₂ + … + ν·f₁₀₀₀
      f₂ fits y − f₁;  f₃ fits y − f₁ − ν·f₂; …
      drop f₁ → the rest correct a base that no longer exists

1. Forest — trees are trained independently on their own bootstrap samples; the ensemble is a plain average, symmetric in every tree: first or five-hundredth makes no difference.
2. Boosting — an additive model with sequential dependency; the first tree carries the largest contribution (a coarse fit of the target), later ones only fine-tune the residuals. Removing f₁ shifts all predictions by roughly its contribution.
3. What the interviewer listens for — independence of trees in bagging vs sequential dependency in boosting; it's the bias/variance question asked point-blank.

⚠️ Common mistake: "one tree out of a thousand either way — no big deal," missing the sequential structure of boosting.

Short answer: It's gradient descent in function space: at each step the new tree approximates the negative gradient of the loss with respect to the ensemble's current predictions (the pseudo-residuals), and the ensemble takes a step F ← F + ν·h. "The tree fits the residuals" is exact only for MSE.

In depth:

rᵢ = −∂L(yᵢ, F(xᵢ))/∂F(xᵢ)       ← pseudo-residuals
hₘ ≈ argmin Σ (rᵢ − h(xᵢ))²      ← the tree fits rᵢ
Fₘ = Fₘ₋₁ + ν·hₘ                 ← descent step with LR ν

MSE:      L = ½(y−F)²  →  r = y − F   (plain residual)
Log-loss: r = y − p,   p = σ(F)       (probability error)

1. Function space — the "parameters" we descend over are the values F(xᵢ) on the training points: the negative gradient says where to move each point's prediction, and the tree generalizes those moves to new points.
2. Why this generalizes — plug in any differentiable loss (quantile, Poisson, ranking) and you get boosting for that task; that's how the objective works in XGBoost/LightGBM/CatBoost.
3. Learning rate — ν is literally the descent step size; smaller step + more trees = more stable.

⚠️ Common mistake: "boosting fits residuals" as the full story. That's the MSE special case; the interviewer wants to hear "negative gradient of the loss w.r.t. the predictions."

Short answer: XGBoost — a second-order approximation (gradient + Hessian) and explicit regularization in the objective. LightGBM — histograms and leaf-wise tree growth: the fastest, but overfits more easily. CatBoost — ordered boosting and ordered target statistics for categoricals: fights target leakage, strong defaults.

In depth:

1. CatBoost's symmetric trees — the same split at every node of a level: a tree is a lookup table of 2^depth leaves, very fast inference and built-in regularization.
2. LightGBM's leaf-wise growth — grows the leaf with the largest gain: converges faster, but on small datasets grows deep unbalanced branches.
3. CIS context — CatBoost is Yandex's product; in interviews at Yandex and its ecosystem, a question about how CatBoost differs is near-guaranteed.

⚠️ Common mistake: "they're all the same, just different speeds." The differences are algorithmic: CatBoost's categorical handling and leakage protection is a distinct idea, not an optimization.

Short answer: It solves prediction shift — a form of target leakage: in standard boosting, an object's residual is computed by a model trained on that very object, and categorical target statistics include the object's own target. CatBoost introduces a random permutation and, for each object, uses statistics and models trained only on the "earlier" objects.

In depth:

1. Leakage in target encoding — replacing a category with the target mean over the whole dataset writes the object's own answer into its feature; on rare categories the feature nearly equals the target → brilliant train, collapse on test.
2. Ordered target statistics — the category statistic for object i is computed only over objects before i in a random permutation (plus a prior): an object's own target never leaks into its feature.
3. Prediction shift in residuals — the same disease in pseudo-residuals: a residual computed by a model that has seen the object is systematically biased. Ordered boosting keeps a set of models: object i's residual comes from a model trained on the permutation prefix before i.
4. The cost — extra computation and memory; in practice CatBoost averages over several permutations.

permutation:  x₃  x₇  x₁  x₅ …
stats and residual for x₅ ← computed only from {x₃, x₇, x₁}

⚠️ Common mistake: hand-rolling target encoding over the full dataset and being surprised by the train/test gap — that's exactly the leak CatBoost fixes out of the box.

Short answer: Lloyd's algorithm: assign points to the nearest centroid, recompute centroids as means, repeat until convergence — minimizing the within-cluster sum of squares (inertia). Choose k via elbow, silhouette, or business logic. It fails on non-spherical clusters, different densities, and unscaled features.

In depth:

for _ in range(max_iter):
    labels = closest_centroid(X, C)        # assignment step
    C = np.array([X[labels == j].mean(0)   # update step
                  for j in range(k)])

1. What we optimize — Σ‖xᵢ − c(xᵢ)‖²; each step never increases the loss → convergence is guaranteed, but only to a local minimum.
2. Choosing k — elbow (the bend in inertia), silhouette (compactness vs separation); often the task dictates k — as many segments as the business can actually serve.
3. Failure modes — elongated/ring-shaped clusters (k-means draws a Voronoi diagram of "spheres"), different sizes and densities, sensitivity to initialization (fixed by k-means++) and to feature scale.
4. Alternatives — DBSCAN for arbitrary shapes, GMM for elliptical clusters and soft memberships.

⚠️ Common mistake: running it on unscaled data — the feature with the largest variance privatizes the distance metric.

Short answer: The principal components are the eigenvectors of the covariance matrix of the centered data (equivalently, the right singular vectors of X from its SVD), ordered by decreasing eigenvalue, i.e. explained variance. They are the orthogonal directions along which the data varies most.

In depth:

center X  →  C = XᵀX/(n−1)
C·vᵢ = λᵢ·vᵢ        ← vᵢ is a component, λᵢ its variance
projection: Z = X·V_k  (first k eigenvectors)

1. The optimization view — the first component maximizes the variance of the projection; each next one does the same subject to orthogonality with the previous. Equivalent formulation: PCA minimizes reconstruction error.
2. In practice — SVD — numerically more stable than explicitly forming the covariance matrix; that's how sklearn implements it.
3. Preprocessing is mandatory — without centering, the first component captures the mean; without scaling, it captures the feature with the largest variance.
4. PCA never looks at the target — it's unsupervised: maximum variance ≠ maximum predictive power, and the signal may live in the minor components.

⚠️ Common mistake: answering "it's dimensionality reduction" with no mechanism. The interviewer wants the words: eigenvectors of the covariance, explained variance, orthogonality.

Short answer: Formally yes, in practice pointless. A sum of linear models is again a linear model, so boosting adds no expressiveness and reduces no bias. Bagging kNN barely changes kNN: it's a stable learner with little variance for averaging to cut.

In depth:

1. Boosting linear models — each step adds wₘᵀx; the total Σwₘᵀx = (Σwₘ)ᵀx is a single linear model you could have trained in one go. The bias of the linear class isn't going anywhere.
2. Bagging needs high variance — averaging reduces variance; for stable learners (linear models, kNN with a reasonable k) predictions barely change from one bootstrap sample to another — there's nothing to average away.
3. Why trees are ideal — a deep tree: low bias / high variance → made for bagging; a shallow one: high bias / low variance → made for boosting.

⚠️ Common mistake: answering "you can't." You can — the question tests whether you understand why ensembles exist at all: bagging eats variance, boosting eats bias.