Data Scientist interview prep

Short answer: Greedily: at each node it scans features and thresholds and picks the split with the largest impurity reduction (Gini or entropy; MSE for regression). With unlimited depth the tree drives leaves to purity — i.e. it memorizes the training set, noise included.

In depth:

1. Split criterion — the gain: impurity(parent) − weighted impurity(children); Gini G = 1 − Σpₖ², entropy −Σpₖ·log pₖ — nearly interchangeable in practice.
2. Greediness — each node is optimized in isolation; the globally optimal tree is NP-hard, so a bad split near the root is never revisited.
3. Why overfitting — the model is piecewise-constant: with unlimited depth every leaf shrinks to a single sample → zero training error, high variance on test.

⚠️ Common mistake: believing the tree finds the globally "best partition" — it's greedy, and that's a fundamental limitation, not an implementation detail.

Short answer: Precision = TP/(TP+FP) — the share of true positives among everything the model flagged as positive. Recall = TP/(TP+FN) — the share of actual positives the model found. Recall is critical where a miss is costly (cancer screening, fraud); precision where a false alarm is costly (spam filters, content takedowns).

In depth:

1. Precision — read the "predicted: 1" row: how many of the model's alerts are real. The cost-of-false-alarm metric.
2. Recall — read the "actual: 1" column: what fraction of real positives we caught. The cost-of-a-miss metric.
3. Trade-off via the threshold — lower the threshold → you catch more positives (recall up) but false alarms grow (precision down). One model is a whole curve of (P, R) pairs, not a single point.

⚠️ Common mistake: reciting the formulas and stopping. A strong answer starts from "which is more expensive — a miss or a false alarm", and only from that decides which metric leads.

Short answer: Scaling matters for models that rely on distances, dot products, or gradient descent: kNN, SVM, k-means, PCA, regularized linear models, neural nets. Trees, forests, and boosting don't need it — they are invariant to monotone transforms.

In depth:

And the key point: the scaler is part of the model. Fit on train only, transform valid/test — otherwise test statistics leak into training.

⚠️ Common mistake: scaling everything "just in case" before the split — a harmless habit turns into leakage; and conversely, wasting time scaling for XGBoost.

Short answer: During training, dropout zeroes random activations with probability p: every batch trains its own “thinned” subnetwork, which prevents neuron co-adaptation and acts like an ensemble of subnetworks. At inference dropout is OFF; to keep activation scale consistent, inverted dropout divides by (1−p) already at training time.

In depth:

1. Mechanics — a binary mask over activations, resampled every forward pass; a neuron cannot rely on specific neighbors and learns more robust features.
2. Interpretation — training an exponential number of weight-sharing subnetworks; inference ≈ averaging that ensemble.
3. Inverted dropout — activations are divided by (1−p) during training, so nothing needs rescaling at inference — the layer simply switches off.
4. Where to put it — usually after fully-connected layers; in convnets it is more often replaced by BatchNorm and augmentations.

drop = nn.Dropout(p=0.3)
model.train()   # mask active, the (1-p) rescaling is built in
model.eval()    # dropout off: deterministic inference

⚠️ Common mistake: forgetting the train/inference difference (and model.eval() in PyTorch): dropout left on at inference produces noisy predictions.

Short answer: An embedding is a dense vector encoding a token's meaning so that semantically similar items sit close in vector space. The key shift: word2vec gives a static vector — one per word, always — while BERT-style models give a contextual one: the vector depends on the sentence, finally solving polysemy.

In depth:

1. word2vec (skip-gram) — learns to predict context words from a word; after training it is a lookup table "word → vector". "Bank" of a river and "bank" with your money get the same vector.
2. Contextual embeddings — a word's vector is computed by a transformer over the whole sentence; "bank" gets different vectors in different phrases.
3. Pretraining objectives differ — skip-gram/CBOW for word2vec vs masked LM for BERT: predict a masked token from bidirectional context.

⚠️ Common mistake: reciting "king − man + woman ≈ queen" but failing to name the real win of contextual embeddings — solving polysemy.