11 Metrics and Class Imbalance Interview Questions and Answers

Short answer: Precision = TP/(TP+FP) — the share of true positives among everything the model flagged as positive. Recall = TP/(TP+FN) — the share of actual positives the model found. Recall is critical where a miss is costly (cancer screening, fraud); precision where a false alarm is costly (spam filters, content takedowns).

In depth:

1. Precision — read the "predicted: 1" row: how many of the model's alerts are real. The cost-of-false-alarm metric.
2. Recall — read the "actual: 1" column: what fraction of real positives we caught. The cost-of-a-miss metric.
3. Trade-off via the threshold — lower the threshold → you catch more positives (recall up) but false alarms grow (precision down). One model is a whole curve of (P, R) pairs, not a single point.

⚠️ Common mistake: reciting the formulas and stopping. A strong answer starts from "which is more expensive — a miss or a false alarm", and only from that decides which metric leads.

Short answer: ROC-AUC is the probability that a randomly chosen positive gets a higher score than a randomly chosen negative. It is a ranking-quality metric, independent of the threshold: 0.5 is random ordering, 1.0 is perfect.

In depth:

1. Probabilistic interpretation — AUC = P(score(x⁺) > score(x⁻)) over random positive–negative pairs. AUC 0.8 = in 80% of such pairs the positive ranks higher.
2. Invariance to monotone transforms — multiply all scores by 2, cube them, pass them through a sigmoid — AUC does not change: only the order matters. Interviewers' favourite follow-up.
3. Says nothing about calibration — AUC does not guarantee that a score of 0.8 means "80% probability": a model can rank perfectly while its probabilities lie shamelessly.

scores, descending:  0.9  0.7  0.6  0.4  0.2
classes:              +    +    −    +    −
(+,−) pairs: 6 total, 5 ordered correctly
→ AUC = 5/6 ≈ 0.83

⚠️ Common mistake: answering "area under the ROC curve" and stopping. The interviewer expects the probabilistic interpretation plus the invariance to thresholds and monotone score transforms.

Short answer: Unknown — first ask about the class balance. With 99% negatives, the constant "predict everything negative" model scores the same 99% accuracy with zero recall. That's the accuracy paradox — a classic screener question.

In depth:

1000 samples: 990 negative, 10 positive
"always 0" model: accuracy = 990/1000 = 99%
                  recall   = 0/10     = 0   ← useless

1. Compare against a baseline — accuracy is only meaningful relative to the majority-class share: 99% when negatives are 99% carries zero information about the model.
2. Per-class metrics — precision and recall on the minority class, the confusion matrix, PR-AUC: they expose a degenerate model instantly.
3. The right move in an interview — a counter-question: "what's the class balance, and what do FP and FN cost?" Asking it is already half the answer.

⚠️ Common mistake: answering "yes, 99% is great". This is a filter question: one second of hesitation before "what's the class balance?" and the interviewer has already drawn the conclusion.

Short answer: PR-AUC. The FPR in the ROC curve is divided by the count of all negatives, which is enormous under imbalance, so even hundreds of false alarms barely move the curve. Precision is divided by the number of things the model flagged, so it exposes the collapse honestly. And the PR-AUC baseline is the prevalence (positive share), not 0.5.

In depth:

Fraud: 100,000 transactions, 100 positives (0.1%). Model: TP = 80, FN = 20, FP = 900.

1. The denominator decides — negatives are so numerous that ROC "forgives" any reasonable volume of FPs; precision only looks at what the model flagged and calls its bluff.
2. Baseline — the random level of ROC-AUC is always 0.5; for PR-AUC it's the prevalence (0.001 here), so a PR-AUC of 0.4 on such data can be a brilliant result.
3. When ROC-AUC is fine — balanced classes, comparing ranking over the whole sample, robustness to prevalence shifts between datasets.

⚠️ Common mistake: bragging about ROC-AUC 0.98 on anti-fraud data. This is THE signature question of fraud and credit-scoring teams — they will immediately ask for precision at the operating recall.

Short answer: F1 = 2PR/(P+R) — the harmonic mean of precision and recall. Harmonic, because it punishes imbalance hard: one high metric cannot compensate for the other one collapsing. F1 is the wrong choice when FP and FN costs differ (use Fβ) or when the business needs calibrated probabilities and direct cost accounting.

In depth:

1. Why not the arithmetic mean — with P = 0.9 and R = 0.1 the arithmetic mean of 0.5 looks tolerable, while F1 = 2·0.9·0.1/1.0 = 0.18 screams failure honestly: one component cannot hide behind the other.
2. Fβ — when recall is β times more important: F2 for screening (a miss is expensive), F0.5 for moderation (a false alarm is expensive).
3. When F1 doesn't fit — asymmetric costs (minimize expected cost directly instead), probability requirements (log-loss, Brier score), model comparison without fixing a threshold (PR-AUC).

⚠️ Common mistake: defaulting to F1 without asking about error costs. F1 implicitly assumes precision and recall are equally valuable — in real tasks they almost never are.

Short answer: The threshold comes from the business costs of errors: minimizing the expected cost of FP and FN, maximizing Fβ, or picking the target point on the ROC/PR curve. 0.5 is an artifact, not a law — just the argmax boundary over probability; under class imbalance and asymmetric costs it is almost never optimal.

In depth:

1. Via cost — predict "1" when p·C_FN > (1−p)·C_FP, which gives threshold = C_FP / (C_FP + C_FN).
2. Mini example — a missed fraud case costs $10,000, a false block $500: threshold = 500 / 10,500 ≈ 0.048. Blocking at 5% confidence is rational.
3. Via a metric — sweep thresholds on validation: maximize Fβ, or fix recall ≥ 0.9 and squeeze the best precision.
4. Operational constraints — "analysts can review 200 alerts a day": the threshold becomes top-k by score, not a probability cutoff at all.

threshold ↓  →  recall ↑, precision ↓
threshold ↑  →  precision ↑, recall ↓
0.5 is a library default, not a solution

⚠️ Common mistake: never questioning 0.5. "Why is your threshold 0.5?" is the standard probe for whether the candidate thinks about the problem or about sklearn's predict().

Short answer: Class weights / cost-sensitive loss (first choice — no data distortion), undersampling, oversampling/SMOTE, threshold moving — and the honest question "is this a data problem at all, or do I just need a different metric". Iron rule: resample only the train fold inside CV, otherwise you leak.

In depth:

1. SMOTE — synthetic points on segments between minority neighbors; on tabular data with categorical features and outliers it often hurts.
2. The key rule — resample strictly inside the train fold of each CV iteration: oversampling before the split scatters copies/interpolations of one object into both train and test → leakage and fantasy metrics.
3. Side effect — resampling changes the train prevalence and breaks probability calibration: after it, scores must be recalibrated.

⚠️ Common mistake: SMOTE before the train/test split. The model "recognizes" the test set through its synthetic twins — the estimate is inflated and everything falls apart in production.

Short answer: A model is calibrated if, among objects predicted p ≈ 0.8, about 80% really are positive. You check it with a reliability diagram and the Brier score, and fix it with Platt scaling or isotonic regression. It's critical wherever the score converts into money: credit scoring, pricing, expected losses.

In depth:

1. Diagnosis — bin the predictions and compare the mean p in each bin with the empirical positive rate:

p bin        actually positive
0.0–0.2      9%
0.4–0.6      44%
0.8–1.0      71%   ← overconfident model: 0.9 ≠ 0.9

2. Who lies out of the box — boosting, SVMs and modern neural nets are systematically miscalibrated (usually overconfident); logistic regression is typically close to calibrated.
3. The fix — Platt scaling (a sigmoid over the scores: little data, S-shaped distortion) or isotonic regression (lots of data, any monotone curve). Calibrate on a separate fold, never on train.
4. Link to imbalance — under/oversampling changes the train prevalence → probabilities are systematically biased; after resampling, calibration is mandatory.

⚠️ Common mistake: "AUC is high, so the probabilities can be trusted." AUC is about order, calibration is about values; knowing the difference is a seniority marker on scoring teams.

Short answer: Linearly: Gini = 2·AUC − 1. It is the same ranking-quality information on a different scale: a random model — AUC 0.5 → Gini 0, a perfect one — AUC 1 → Gini 1.

In depth:

Gini = 2·AUC − 1
AUC 0.50 → Gini 0.0   (random ranking)
AUC 0.75 → Gini 0.5
AUC 1.00 → Gini 1.0
range: [−1, 1]; in real credit scoring
a good Gini ≈ 0.3–0.6 (AUC ≈ 0.65–0.8)

1. Geometry — Gini is the ratio of the area between the ROC curve and the diagonal to the area of the ideal triangle above the diagonal; hence the linear formula.
2. Bank-scoring vocabulary — Sber, T-Bank and credit-risk teams everywhere say "Gini", not "AUC": model monitoring, validation reports, degradation alerts — all quoted in Gini.
3. Practical meaning — nothing beyond AUC: same ranking, same invariances. "Gini dropped from 0.45 to 0.38" is a standard model-monitoring phrase you are expected to parse instantly.

⚠️ Common mistake: interviewing at a bank without knowing Gini and AUC are the same thing up to a linear transform. For a scoring team this is baseline vocabulary, not a bonus question.

Short answer: RMSE is minimized by the conditional mean and penalizes large errors quadratically — outliers dominate it. MAE is minimized by the median — robust. MAPE is undefined at zero actuals and asymmetric: over-forecasts are penalized without bound, under-forecasts by at most 100%, so optimizing MAPE biases forecasts low. For asymmetric business costs — quantile loss.

In depth:

1. RMSE vs MAE — the question is "what hurts more: the average miss or the rare catastrophe". Demand with rare spikes: RMSE forces the model to "insure" against them.
2. MAPE — divides by the actual: explodes at zero demand; an over-forecast error is unbounded (>100% happens), an under-forecast is capped at 100%.
3. Asymmetric costs — lost sales costlier than excess stock? Quantile (pinball) loss with τ > 0.5 is the demand-forecasting standard.

⚠️ Common mistake: optimizing MAPE on demand data with zeros and spikes: the metric blows up at zeros and systematically rewards under-forecasting.

Short answer: All four evaluate the quality of the top of a ranked list, not classification. Precision@k — the share of relevant items in the top k; MRR — how high the first relevant item sits; MAP — precision averaged over the positions of all relevant items; NDCG — graded relevance with a logarithmic position discount, normalized by the ideal ranking.

In depth:

1. Precision@k — the user only sees the first screen: count relevant items in the top k, the tail doesn't matter.
2. MRR — 1/rank of the first relevant item, averaged over queries: for "one right answer" scenarios (search, autocomplete, QA).
3. MAP — mean average precision over queries: accounts for both completeness and order, but relevance is binary only.
4. NDCG — graded relevance (0–3), position i contributes rel/log2(i+1), and the sum is divided by the DCG of the ideal ordering.

ranking (rel):   [2, 0, 3, 1]
DCG  = 2/log2(2) + 0/log2(3) + 3/log2(4) + 1/log2(5)
     = 2 + 0 + 1.5 + 0.43 = 3.93
ideal [3, 2, 1, 0]: IDCG = 3 + 1.26 + 0.5 + 0 = 4.76
NDCG = 3.93 / 4.76 ≈ 0.83

5. When ranking metrics, not classification ones — search and recommendations: relevance isn't binary and position decides everything. A staple question at Yandex, Avito, Ozon.

⚠️ Common mistake: evaluating a recommender with a global ROC-AUC over all user–item pairs: it can't see that the user is shown only the top of the list, where position is everything.