12 ML Model Serving Interview Questions and Answers

Short answer: Batch inference computes predictions ahead of time on a schedule and stores them; online inference answers each request in real time. The choice comes down to two questions: are the inputs known in advance, and how quickly do predictions go stale?

In depth:

The standard hybrid: batch-precompute candidates and embeddings, then run only a light reranker online using fresh request-time features.

⚠️ Common mistake: moving everything online "for freshness." If a prediction can be computed ahead of time, batch is almost always cheaper, simpler, and more reliable.

Short answer: Batching raises GPU throughput at the cost of per-request latency: requests sit in a queue while the batch fills up. SLAs are stated on p99, not the mean — the tail is what suffers first.

In depth:

1. GPUs are throughput devices — a forward pass at batch=1 uses single-digit percent of available FLOPS; the matrix units idle, money burns.
2. Numeric example — forward pass: batch=1 → 10 ms, batch=32 → 40 ms. Throughput grows from 100 to 800 RPS (8x), but the request that arrived first waits for the batch to fill: latency creeps from 10 ms to 40+ ms.
3. p99, not the mean — the average can look fine while requests caught in an unlucky batch cycle blow through the SLA.

batch=1 : 10 ms/forward → 100 RPS, latency ~10 ms
batch=32: 40 ms/forward → 800 RPS, latency 40+ ms (queueing!)

⚠️ Common mistake: optimizing mean latency. Users and SLAs live at p99 — with batching, the tail grows first.

Short answer: The server queues incoming requests and assembles them into one batch: either max_batch_size is reached or max_queue_delay expires — then the batch goes through a single forward pass. Clients send one request at a time; the GPU sees batches.

In depth:

1. Two knobs — max_batch_size (size ceiling) and max_queue_delay_microseconds (how long to wait for stragglers). The delay is a direct surcharge on every request's latency.
2. What you gain — GPU utilization and throughput grow severalfold with zero client-side changes.
3. The cost is the tail — under bursty traffic a request can sit out the full delay and land in a large batch: p99 grows noticeably faster than p50.
4. When NOT to enable it — a tight p99 budget with no slack; a runtime optimized for batch=1; traffic so sparse that batches never fill — then the delay is pure loss.

requests ──► queue ──► [32 collected OR 5 ms elapsed] ──► one forward pass

⚠️ Common mistake: cranking max_queue_delay for a pretty throughput benchmark — silently gifting every production request extra milliseconds of latency.

Short answer: gRPC is the standard on the internal model path: HTTP/2, binary protobuf serialization (cheap on tensor payloads), native streaming, and connection multiplexing. REST is kept for external clients and debugging.

In depth:

The difference matters when serialization is comparable to the inference itself: a small model, large tensors, thousands of RPS.

⚠️ Common mistake: "switch to gRPC and it'll get faster." If the model takes 200 ms, saving a couple of milliseconds on transport solves nothing. Transport matters with millisecond-level inference and fat payloads — knowing exactly when is the real answer.

Short answer: The offline contour (warehouse/Parquet) serves features for training — large historical datasets with point-in-time-correct joins. The online contour (Redis/KV) serves the same features for inference in milliseconds. The point is that a single feature definition materializes into both contours — that is what kills training-serving skew.

In depth:

1. Offline — bulky historical slices; the key property is point-in-time correctness: the feature's value as of the event, with no leakage from the future.
2. Online — a key-value store with single-digit-millisecond reads by entity key (user_id, item_id).
3. One definition — the feature is written once (SQL/DSL); pipelines materialize it into both the warehouse and the KV store: training and serving see the same logic.
4. Per-feature freshness SLA — "purchases over 30 days" can refresh daily, "clicks in the last 5 minutes" only via streaming; the freshness budget is set per feature.

                   ┌─► offline store (Parquet) ─► training (point-in-time joins)
feature definition ─┤
                   └─► online store (Redis/KV) ─► inference (< 10 ms)

⚠️ Common mistake: treating a feature store as "a database for features." Its value is not storage but the single feature definition from which both contours are built.

Short answer: With a per-hop latency budget. Then: split candidate generation from ranking, fetch features in a single batched call, cache aggressively, scale out with stateless replicas behind a LB, and agree on graceful degradation up front. 10k QPS is about the architecture around the model, not the model.

In depth:

p99 budget, 100 ms:
  LB + network             ~5 ms
  features from online store ~15 ms (Redis mget, one RTT!)
  candidate gen (ANN)      ~15 ms (top-500)
  ranker forward pass      ~40 ms (batch of 500 candidates)
  serialization + response ~10 ms
  tail headroom            ~15 ms

1. Two stages — cheap retrieval narrows millions down to hundreds; the heavy ranker scores only those.
2. Features in one call — mget/pipeline; N sequential RTTs would eat the whole budget.
3. Caching — features of hot entities and precomputed scores for popular queries.
4. Replicas — stateless serving behind a LB, autoscaling on QPS and p99.
5. Degradation — ranker timeout → return popularity-ordered candidates or cached scores, not a 500.

⚠️ Common mistake: spending the entire budget on the model and "remembering" the network RTTs for feature fetches later — they easily eat 20-30 ms.

Short answer: Reduce the work per request: distill or quantize the model, build a cascade (a light model filters, the heavy model scores the top-k), precompute and cache popular entities, return async partial responses. "Buy more GPUs" is not an option: replicas fix throughput, not single-request latency.

In depth:

⚠️ Common mistake: proposing horizontal scaling. A single request still goes through one 200 ms forward pass — replicas add parallelism, not speed.

Short answer: HNSW is a multi-layer navigable-small-world graph: search starts on the sparse upper layers, greedily descending toward the query, and finishes on the dense bottom layer — roughly O(log n). Best recall at low latency, but the graph lives in RAM. IVF saves memory; brute force is exact and sufficient below ~1M vectors.

In depth:

1. HNSW — upper layers act as "highways" for long jumps, the bottom layer does precise local navigation; recall is tuned via efSearch: higher means more accurate and slower.
2. IVF — the corpus is clustered (k-means); at query time you scan only the nprobe nearest lists. Cheaper on memory; recall dips near cluster boundaries.
3. Flat (brute force) — an honest scan of all vectors: 100% recall; up to ~1M vectors on a GPU it is often faster than people assume.

The choice is a recall-latency-memory triangle: fix two, pay with the third.

⚠️ Common mistake: reaching for HNSW at 100k vectors "because everyone does" — there brute force is more accurate, simpler, and already fast enough.

Short answer: Three layers: a feature cache (TTL set by feature volatility), a prediction cache (key = entity + model version), an embedding cache (embeddings change only on retrain — long TTL). The key question for any cache in an ML system is not hit rate but what exactly invalidates it.

In depth:

1. TTL from volatility — demographics live for days; a 5-minute click counter is barely cacheable at all.
2. Model version in the key — rolling out a new version automatically invalidates the prediction cache; otherwise part of the traffic silently gets the old model's scores.
3. Event-driven invalidation — profile updated → evict its features without waiting for the TTL.

⚠️ Common mistake: caching predictions built on fast-changing features: a fraud score computed before the suspicious transaction looks perfectly "fresh" coming out of the cache.

Short answer: A versioned artifact plus its lineage: data hash, git commit, config, offline-eval metrics. The path to prod is stage transitions (staging → production); rollback is repointing an alias to the previous version, with no rebuild.

In depth:

1. Artifact + lineage — for any prod version you can reconstruct: which data it trained on (data hash), which code (commit), which config, and what metrics it posted.
2. Stages and aliases — staging → smoke tests and shadow → production; serving pulls the model by alias (prod), never by version number.
3. Rollback — repoint the alias to the previous version: seconds, no service redeploy.
4. Model and features are versioned together — a new model version expects a different feature set; a mismatch between feature config and artifact is the classic silent breakage.

train run ─► registry: v42 + lineage (data hash, commit, config, metrics)
v42: staging ──smoke/shadow──► prod (alias)
rollback = repoint alias prod → v41

⚠️ Common mistake: "our model sits in S3." A .pt file with no lineage and no stages is a red flag to the interviewer: such a model can be neither reproduced nor safely rolled back.

Short answer: A ladder: shadow (duplicate the request to the new version asynchronously, never return its result — validating latency and score sanity) → canary at 1-5% → wider rollout with a sticky split (consistent hashing on user id). Each rung has exit metrics and a pre-written rollback trigger.

In depth:

shadow (0% of decisions) ─► canary 1–5% ─► 25% ─► 100%
        │                       │           │
        └───────────────────────┴───────────┴─► rollback: alias to v-1

1. Shadow — the new version sees live traffic, its responses are only logged; latency spikes, errors, and wild score distributions get caught before the first user does.
2. Canary — 1-5% of live decisions; compare system metrics and scores against control.
3. Sticky split — consistent hashing on user id, not a random pick per request: a user must not bounce between versions.
4. Logging hygiene — under shadow, features get logged twice; fail to tag the source and you double your training logs.

Product impact is measured by an A/B experiment — a separate mechanism; this ladder is purely about infrastructural rollout safety.

⚠️ Common mistake: a random per-request split — the same user gets old scores one moment and new ones the next, and debugging turns into hell.

Short answer: Since p50 has not moved, the typical forward pass is fine — it is the tail that degraded. The suspects are tail-only: batching queue saturation, a cold cache for rare entities, one sick replica, GC pauses, thread-pool exhaustion. Per-hop tracing first — the model is the last thing you touch.

In depth:

Triage order:

1. Per-hop tracing — where exactly did the tail grow: queue, feature fetch, forward pass, network? Without a distributed trace, everything else is guessing.
2. One replica — p99 per replica, individually: one sick machine (throttling, noisy neighbor) ruins the aggregate tail while p50 stays healthy.
3. Batching queue — the deploy changed max_queue_delay or traffic grew: some requests wait for the batch to fill, a pure tail hit.
4. Cold cache — the deploy flushed the feature/prediction cache: hot keys re-warm instantly (p50 fine), rare entities keep missing the cache (p99 climbs).
5. Runtime — GC pauses, thread-pool or connection-pool exhaustion: rare but long stalls — literally the portrait of p99.

⚠️ Common mistake: starting with "the model got slower." If the forward pass had slowed down, p50 would have moved too — the symptom itself rules out the model as observation number one.