9 ML Engineering ML Engineering Interview Questions and Answers

Short answer: Separate the deterministic from the stochastic. Transforms, losses and metrics get ordinary unit tests on hand-computable cases; training is covered by smoke tests (overfit a tiny batch, shape and gradient-flow checks) and behavioral tests with tolerances.

In depth:

1. Test the code, not the model — deterministic pieces (features, losses) deserve exact asserts; stochastic ones get tolerances and invariants.
2. Smoke test in CI — a short 2–3 step run: the pipeline assembles, the loss is finite and decreasing.

⚠️ Common mistake: "we look at test accuracy" — that's model evaluation, not code testing. A dataloader bug can cost a couple of metric points and never surface.

Short answer: Go from data to architecture, not the other way around. First eyeball a batch and its labels, then run the decisive experiment — overfit a single batch. Only then LR, training-loop wiring bugs, and initialization.

In depth:

1. Eyeball the data: a raw batch, the labels, X ↔ y alignment
2. Overfit ONE batch to ~100% — the decisive experiment:
   can't → the wiring is broken; can → data or LR
3. LR sweep: ×10 up and down — a too-large LR
   also looks like "loss won't go down"
4. Wiring: forgotten optimizer.zero_grad(),
   wrong loss reduction, labels misaligned with logits
   (shift, class order), frozen parameters,
   missing model.train()
5. Initialization and input normalization

1. The order matters more than the list — the interviewer is grading exactly this: you check data before architecture, because data breaks orders of magnitude more often.
2. The one-batch overfit — a single check that bisects the hypothesis space: failing to memorize 10 examples rules out "the task is hard" as an excuse.

⚠️ Common mistake: reaching for architecture changes or extra layers first — it's almost always the data or the loop wiring, not model capacity.

Short answer: Schema (columns and dtypes), value bounds, null fractions, categorical domains, row count relative to the last run, and the label distribution. The checks are code in the pipeline that fails loudly BEFORE training — not an analyst's eyes after.

In depth:

def validate(df, ref):
    assert set(df.columns) == set(ref.columns)         # schema
    assert df["age"].between(18, 100).all()            # value bounds
    assert df["income"].isna().mean() < 0.02           # null fraction
    assert set(df["region"]) <= ref.region_domain      # categorical domain
    assert 0.5 < len(df) / ref.row_count < 2.0         # volume vs last run
    assert abs(df["y"].mean() - ref.y_rate) < 0.05     # label distribution

1. Fail loudly vs quarantine — a schema or label violation kills the whole pipeline; isolated broken rows can go to quarantine with an alert, but the allowed fraction is itself a check.
2. Reference from the previous run — half the checks are relative: comparison against pinned statistics of the last successful run.
3. Tools — Great Expectations, pandera; but the interviewer cares about the concept of "validation as a gate", not the library name.

⚠️ Common mistake: validating only the schema. The dtypes match, but a column silently became 40% null — training will "succeed" on garbage.

Short answer: The seed only controls random number generators. What remains: non-deterministic CUDA kernels, cuDNN autotuning, dataloader worker scheduling, non-associative float arithmetic, and library versions. The full answer: seeds + deterministic flags + a pinned environment + versioned data/code/config.

In depth:

1. The cost of determinism — deterministic kernels are noticeably slower; in production teams often pin everything except them and accept noise within a tolerance.
2. Reproducible ≠ bit-exact — the practical standard: the same metric within a stated spread, with full lineage recoverable.

⚠️ Common mistake: "I set seed=42, so it's reproducible" — on a different GPU or cuDNN version the numbers will diverge.

Short answer: Everything the result is built from: the code's git commit, the data version/hash, the full config with hyperparameters, the environment, metrics over steps, and artifacts (checkpoints). Logging metrics alone is the trap: you can't reproduce the winner afterwards.

In depth:

1. Code — the exact commit hash plus a dirty-working-tree flag: an uncommitted patch makes the commit useless.
2. Data — a dataset version or hash (DVC, snapshot): "the same table" a month later is different data.
3. The whole config — every hyperparameter, split, feature version; not just the knobs you turned in this experiment.
4. Environment — a dependency lock file or image, CUDA/driver version.
5. Metrics over steps — train/val curves, not one final number: divergence and early stopping become visible.
6. Artifacts — the checkpoint, tokenizer, preprocessor — attached to the run.

run = code (commit) + data (hash) + config + env
      └─► metrics over steps + artifacts (checkpoint)

⚠️ Common mistake: comparing runs by metric when they silently used different data or splits — the comparison is meaningless without identical lineage.

Short answer: It's a producer-consumer problem: the GPU is almost always starved by the input pipeline. Profile first (torch.profiler, py-spy, a sawtooth in nvidia-smi), then fix: num_workers, pin_memory, offline preprocessing, prefetch — and remove synchronizations like a per-step .item().

In depth:

1. Profile, don't guess: torch.profiler / py-spy;
   a sawtooth in nvidia-smi utilization = GPU waiting for data
2. Dataloader: num_workers > 0, pin_memory=True,
   .to(device, non_blocking=True)
3. Move heavy work out of __getitem__: pre-tokenize,
   pre-resize offline, cache on fast storage
4. Prefetch the next batch; a larger batch
   if memory allows
5. Synchronizations in the loop: .item() / logging
   every step, stray .cpu() — hidden sync points
6. Storage: network disks, millions of tiny files →
   sharded formats (webdataset, tfrecord)

1. The framing — a conveyor "disk → CPU decode → H2D copy → GPU": 30% utilization means the slowest stage isn't the GPU.
2. The profiler settles the argument in minutes — the timeline shows exactly who holds the step: DataLoader, the copy, or compute.

⚠️ Common mistake: asking for a bigger GPU first — that speeds up the consumer, which is already idle 70% of the time.

Short answer: num_workers spawns separate processes (not threads — because of the GIL) that read and decode batches in parallel with prefetching. pin_memory places the ready batch in page-locked memory, which enables asynchronous copies to the GPU via non_blocking=True.

In depth:

loader = DataLoader(ds, batch_size=64,
    num_workers=8,     # 8 processes decode and prefetch batches
    pin_memory=True)   # batches land in page-locked (pinned) memory

x = batch.to("cuda", non_blocking=True)  # async H2D:
# the copy overlaps with compute — only from pinned memory

1. Processes, not threads — the Python GIL prevents parallel decoding in threads; workers are forked and communicate through a queue.
2. Pinned memory — the OS can't page it out, so the DMA copy to the GPU runs without a staging buffer and asynchronously.
3. The cost of workers — each holds its own copy of the dataset in memory: RAM ×N; lazy initialization of heavy objects is mandatory.

⚠️ Common mistake: "set num_workers=32, it'll be faster" — more workers than CPU cores means context switching and degradation; tune by measurement, starting from the core count.

Short answer: Each step is idempotent, exchanges only explicit versioned artifacts with its neighbors, is parametrized by the run date (for backfills), and survives retries. Plus two gates: data validation BEFORE training and eval BEFORE promotion to the registry.

In depth:

ingest ──► validate ──► features ──► train ──► eval ──► promote
              │ gate: fail            │          │ gate: metrics
              ▼ before training       ▼          ▼ ≥ threshold
            stop               checkpoint     registry

every step: idempotent · reads/writes versioned
artifacts · parametrized by date · retry-safe

1. Idempotency — re-running a step for the same date yields the same result and duplicates nothing: write to a versioned/dated path, don't append.
2. Explicit artifacts — steps talk through storage (parquet, a checkpoint), not process memory: any step can be re-run in isolation.
3. Date parametrization — run(date) instead of "today": a month-long backfill is just 30 runs.
4. Gates — broken data never reaches training, a weak model never reaches the production registry; promotion always compares against the current champion.

⚠️ Common mistake: listing Airflow operator names instead of step properties — the interviewer is asking about invariants; the orchestrator is secondary.

Short answer: Before saying "drift", rule out code and data divergence between training and serving: two feature code paths, point-in-time violations in the training join, mismatched preprocessing versions, stale online features, serialization losses.

In depth:

1. The delta vs the data scientist's answer — "drift" is a statistical hypothesis; the senior engineering answer first names the code paths where training and prod see different data.

⚠️ Common mistake: retraining "on fresh data" right away — if serving computes the features differently, retraining fixes nothing.