9 Data Analyst Experimentation Interview Questions and Answers

Short answer: An A/B test is a randomized experiment: users are split at random into control and treatment, shown different versions, and compared on a single pre-chosen primary metric under a fixed decision rule.

In depth:

1. Hypothesis — what you change, which metric it should move, and why.
2. Metric and MDE — one primary metric plus a minimum detectable effect (MDE).
3. Randomization — a random, stable split into control/treatment.
4. Sample size and duration — compute and fix them before launch.
5. Run to N — run until the sample is reached, without peeking at interim results.
6. Analysis — compare the metric, compute the p-value and the confidence interval of the lift.
7. Decision — ship or roll back per the pre-defined rule.

Hypothesis → Randomize → Run to N → Analyze → Decision
              │                                  │
       control / treatment              ship ✓ / rollback ✗

⚠️ Common mistake: changing the UI mid-test or stopping the moment it first looks 'significant' — both break the conclusions.

Short answer: Sample size is driven by the baseline rate, the minimum detectable effect (MDE), power (1−β, usually 80%), and the significance level α (usually 5%): the smaller the MDE, the more data you need. You fix duration up front to rule out peeking and to cover whole weekly cycles.

In depth:

• Baseline rate — rare events have relatively higher variance → need more sample.
• MDE — the strongest lever: sample grows roughly as 1/MDE².
• Power (1−β) — chance of detecting a real effect; 80% is standard.
• α — false-positive threshold; a smaller α means a larger sample.
• Duration — divide N by daily traffic and round up to whole weeks.

# Sample size per group: difference of proportions
from scipy.stats import norm
z_a, z_b = norm.ppf(0.975), norm.ppf(0.80)   # α=5% (two-sided), power 80%
p1, p2 = 0.10, 0.11                          # baseline 10%, MDE = +1 pp
var = p1*(1-p1) + p2*(1-p2)
n = (z_a + z_b)**2 * var / (p2 - p1)**2
print(round(n))   # ≈ 14,700 per group

⚠️ Common mistake: not fixing duration in advance and 'stopping once it's significant' — that is peeking, which inflates false positives.

Short answer: Statistical significance (p < α, or the confidence interval of the lift does not cross 0) says the effect is unlikely to be chance; practical significance says its size is worth shipping. A non-significant result means 'not enough data to tell the effect from zero,' not 'there is definitely no effect.'

In depth:

• Statistical significance — p below α, or the 95% CI of the lift does not cross 0.
• Practical significance — the effect clears the payoff threshold (MDE/business sense).
• Confidence interval of the lift — more informative than a bare p: shows sign, magnitude, and uncertainty.
• Non-significant ≠ no effect — it's either a truly null effect or insufficient power.

Lift and its 95% CI   (│ = zero)
            │
A)  [-------│---●----]    CI [−0.5%, +1.8%] → crosses 0 → NOT significant
            │
B)          │  [---●---]  CI [+0.7%, +2.1%] → doesn't cross 0 → significant
            │

⚠️ Common mistake: declaring 'no effect' from a non-significant result — say instead 'we did not detect an effect of size ≥ MDE.'

Short answer: Peeking is repeatedly checking significance while the test runs and stopping the moment p < 0.05. Each extra look is another chance at a fluke 'win,' so the real α climbs well above 5%. Many metrics/variants cause the same multiple-comparisons problem and need correction.

In depth:

1. Why peeking is dangerous — with many looks, the chance of crossing the threshold at least once approaches 100%.
2. How to do it right — fix N and look once; if you need interim looks, use sequential testing / alpha-spending.
3. Multiple comparisons — k metrics at α=5% give a risk ≈ 1−0.95^k; apply Bonferroni or control FDR (Benjamini–Hochberg).

⚠️ Common mistake: stopping on day one when p happened to dip below 0.05.

Short answer: One primary metric decides the test; guardrail metrics make sure you didn't break something important (revenue, speed, churn); secondary metrics help explain the mechanism. The more 'win' metrics you add, the higher the chance of a fluke significance, so keep the set short.

In depth:

• Primary — chosen in advance, drives the decision; exactly one.
• Guardrail — must not degrade; blocks the ship even if the primary goes up.
• Secondary — for interpretation and hypotheses, not for the decision.

⚠️ Common mistake: loading up a dozen 'win' metrics and declaring success on whichever happened to light up — that's hidden multiple comparisons.

Short answer: Validity is broken by an imbalance in the group split (SRM — sample ratio mismatch), novelty/primacy effects, network effects (treatment affecting control), and seasonality. SRM is a red flag of a broken experiment: don't trust the results until you've found the cause of the discrepancy.

In depth:

• SRM — the actual split differs significantly from the expected one (test with χ²); a sign of a bug in randomization, logging, or filtering.
• Novelty/primacy — response shifts over time: people either poke a novelty out of curiosity or reject it out of habit.
• Network effects — groups aren't isolated (social, marketplaces), the effect 'leaks.'
• Seasonality — day of week, promos, holidays distort short tests.

⚠️ Common mistake: seeing a 'significant' result and not checking SRM — with skewed groups the comparison is invalid to begin with.

Short answer: An aggregate result can flip inside segments (Simpson's paradox) when segments differ a lot in size and respond differently. But slicing after the fact into dozens of segments is hidden multiple comparisons: something 'significant' will turn up by chance.

In depth:

• Simpson's paradox — the effect's direction in every segment is opposite to the total because of an imbalanced split across groups (often a symptom of SRM).
• Slicing to significance — the more cuts, the higher the chance of a false discovery; fix decision segments in advance.
• What to do — pre-register key segments; treat the rest as hypothesis generation, not a conclusion.

Treatment wins in each segment yet loses overall: the groups are skewed across platforms.

⚠️ Common mistake: slicing the data until some segment shows p < 0.05 and passing that off as the result.

Short answer: When user-level randomization is impossible (strong network effects, marketplaces, price or geo changes), use quasi-experiments: switchback, difference-in-differences, synthetic control, geo holdouts. They give weaker causal claims than an RCT and rely on extra assumptions.

In depth:

⚠️ Common mistake: treating a quasi-experiment as a full RCT — you must check its assumptions (e.g. parallel trends for diff-in-diff).

Short answer: Randomization, on average, balances the groups on all factors — known and unknown — so the difference in the metric can be attributed to the change itself, not to hidden confounders. Observational analysis can't do this: correlation ≠ causation. The price of that rigor is time, traffic, and engineering effort.

In depth:

1. Confounder — a shared factor Z that affects both the 'feature' X and the metric Y, creating a spurious link.
2. Randomization removes it — random assignment cuts the Z→X link, balancing Z across groups.
3. Observationally — Z is uncontrolled; the visible X–Y link may be entirely due to Z.
4. The price — you need sample, weeks of waiting, and infrastructure for a correct split.

Observational:            Z (confounder)
                        ↙            ↘
                     X ······?······ Y   link may run through Z

Randomized:    coin flip → X           Z balanced across groups
                           ↘
                            → Y        the X→Y arrow is 'clean'

⚠️ Common mistake: drawing causal conclusions from observational correlations without closing confounders via randomization or at least controls.