9 Data Analyst Experimentation Interview Questions and Answers

Short answer: An A/B test is a randomized experiment: users are split at random into control and treatment, shown different versions, and compared on a single pre-chosen primary metric under a fixed decision rule.

In depth:

1. Hypothesis — what you change, which metric it should move, and why.
2. Metric and MDE — one primary metric plus a minimum detectable effect (MDE).
3. Randomization — a random, stable split into control/treatment.
4. Sample size and duration — compute and fix them before launch.
5. Run to N — run until the sample is reached, without peeking at interim results.
6. Analysis — compare the metric, compute the p-value and the confidence interval of the lift.
7. Decision — ship or roll back per the pre-defined rule.

Hypothesis → Randomize → Run to N → Analyze → Decision
              │                                  │
       control / treatment              ship ✓ / rollback ✗

⚠️ Common mistake: changing the UI mid-test or stopping the moment it first looks 'significant' — both break the conclusions.

Short answer: Sample size is driven by the baseline rate, the minimum detectable effect (MDE), power (1−β, usually 80%), and the significance level α (usually 5%): the smaller the MDE, the more data you need. You fix duration up front to rule out peeking and to cover whole weekly cycles.

In depth:

• Baseline rate — rare events have relatively higher variance → need more sample.
• MDE — the strongest lever: sample grows roughly as 1/MDE².
• Power (1−β) — chance of detecting a real effect; 80% is standard.
• α — false-positive threshold; a smaller α means a larger sample.
• Duration — divide N by daily traffic and round up to whole weeks.

# Sample size per group: difference of proportions
from scipy.stats import norm
z_a, z_b = norm.ppf(0.975), norm.ppf(0.80)   # α=5% (two-sided), power 80%
p1, p2 = 0.10, 0.11                          # baseline 10%, MDE = +1 pp
var = p1*(1-p1) + p2*(1-p2)
n = (z_a + z_b)**2 * var / (p2 - p1)**2
print(round(n))   # ≈ 14,700 per group

⚠️ Common mistake: not fixing duration in advance and 'stopping once it's significant' — that is peeking, which inflates false positives.

Short answer: Statistical significance means p < α under a prespecified valid procedure: assuming the null hypothesis and test assumptions, data at least as extreme are unlikely. It is not the probability that the effect is “due to chance” or that the null is true. Practical significance asks whether benefits justify costs and risks.

In depth:

1. Effect and interval — report the estimate, units, and confidence interval. For a corresponding two-sided 5% test, excluding zero from a 95% interval agrees with rejecting the null; one-sided and adjusted procedures require matching intervals.
2. MDE — the effect the design is powered to detect at a specified power and α. It is a planning parameter, not an automatic business-value threshold or a bound on effects excluded by the observations.
3. Non-significance — the null was not rejected; limited precision or a small effect are both possible. Do not conclude “we only need more data” or “effects ≥ MDE are absent.” Inspect the values still compatible with the interval.
4. Practical decision — prespecify a minimum worthwhile effect and guardrails. To establish the absence of a practically important difference, use an appropriate equivalence design with predefined margins.

For example, a lift interval of [−0.5, +1.8] percentage points includes zero and a meaningful positive effect; it does not justify “there is no effect.”

⚠️ Common mistake: treating p ≥ α as proof of zero effect, or comparing only p-values. Discuss magnitude, uncertainty, design, and business thresholds together.

Short answer: Peeking is repeatedly checking significance while the test runs and stopping the moment p < 0.05. Each extra look is another chance at a fluke 'win,' so the real α climbs well above 5%. Many metrics/variants cause the same multiple-comparisons problem and need correction.

In depth:

1. Why peeking is dangerous — with many looks, the chance of crossing the threshold at least once approaches 100%.
2. How to do it right — fix N and look once; if you need interim looks, use sequential testing / alpha-spending.
3. Multiple comparisons — k metrics at α=5% give a risk ≈ 1−0.95^k; apply Bonferroni or control FDR (Benjamini–Hochberg).

⚠️ Common mistake: stopping on day one when p happened to dip below 0.05.

Short answer: One primary metric decides the test; guardrail metrics make sure you didn't break something important (revenue, speed, churn); secondary metrics help explain the mechanism. The more 'win' metrics you add, the higher the chance of a fluke significance, so keep the set short.

In depth:

• Primary — chosen in advance, drives the decision; exactly one.
• Guardrail — must not degrade; blocks the ship even if the primary goes up.
• Secondary — for interpretation and hypotheses, not for the decision.

⚠️ Common mistake: loading up a dozen 'win' metrics and declaring success on whichever happened to light up — that's hidden multiple comparisons.

Short answer: Validity is broken by an imbalance in the group split (SRM — sample ratio mismatch), novelty/primacy effects, network effects (treatment affecting control), and seasonality. SRM is a red flag of a broken experiment: don't trust the results until you've found the cause of the discrepancy.

In depth:

• SRM — the actual split differs significantly from the expected one (test with χ²); a sign of a bug in randomization, logging, or filtering.
• Novelty/primacy — response shifts over time: people either poke a novelty out of curiosity or reject it out of habit.
• Network effects — groups aren't isolated (social, marketplaces), the effect 'leaks.'
• Seasonality — day of week, promos, holidays distort short tests.

⚠️ Common mistake: seeing a 'significant' result and not checking SRM — with skewed groups the comparison is invalid to begin with.

Short answer: An aggregate result can flip inside segments (Simpson's paradox) when segments differ a lot in size and respond differently. But slicing after the fact into dozens of segments is hidden multiple comparisons: something 'significant' will turn up by chance.

In depth:

• Simpson's paradox — the effect's direction in every segment is opposite to the total because of an imbalanced split across groups (often a symptom of SRM).
• Slicing to significance — the more cuts, the higher the chance of a false discovery; fix decision segments in advance.
• What to do — pre-register key segments; treat the rest as hypothesis generation, not a conclusion.

Treatment wins in each segment yet loses overall: the groups are skewed across platforms.

⚠️ Common mistake: slicing the data until some segment shows p < 0.05 and passing that off as the result.

Short answer: Inability to randomize individual users does not rule out a randomized experiment. Switchback designs can randomly assign treatment to time blocks, and geo tests can randomize regions. Difference-in-differences and synthetic control usually use observational comparisons and need additional assumptions for causal interpretation.

In depth:

1. Identify the assignment unit and randomization scheme first. The name “switchback” or “geo holdout” does not by itself establish randomization.
2. Network effects and effects persisting over time can violate independence; washout reduces but does not automatically eliminate carryover.
3. Do not treat every user as an independent unit when assignment is by region or time block. Uncertainty must account for the design.

⚠️ Common mistake: calling every alternative to user-level A/B a quasi-experiment. Distinguish randomization from observational identification and justify the causal claim through specific assumptions.

Short answer: Randomization, on average, balances the groups on all factors — known and unknown — so the difference in the metric can be attributed to the change itself, not to hidden confounders. Observational analysis can't do this: correlation ≠ causation. The price of that rigor is time, traffic, and engineering effort.

In depth:

1. Confounder — a shared factor Z that affects both the 'feature' X and the metric Y, creating a spurious link.
2. Randomization removes it — random assignment cuts the Z→X link, balancing Z across groups.
3. Observationally — Z is uncontrolled; the visible X–Y link may be entirely due to Z.
4. The price — you need sample, weeks of waiting, and infrastructure for a correct split.

Observational:            Z (confounder)
                        ↙            ↘
                     X ······?······ Y   link may run through Z

Randomized:    coin flip → X           Z balanced across groups
                           ↘
                            → Y        the X→Y arrow is 'clean'

⚠️ Common mistake: drawing causal conclusions from observational correlations without closing confounders via randomization or at least controls.