12 ML Engineering LLM Inference Interview Questions and Answers

Short answer: The KV-cache stores the per-layer K and V matrices of every token processed so far. Without it, autoregressive generation would recompute K and V for the whole prefix at every step — O(n²) work per sequence; with the cache, each decode step computes K, V and Q only for the single new token — O(n) total.

In depth:

1. Why it works — K and V of past tokens do not depend on future ones: compute them once, reuse them at every later step.
2. What is cached — only K and V, separately per layer. The current token's Q is computed fresh: it is needed exactly once — for this token to attend over the past — and is never reused.
3. The cost — memory: the cache grows linearly with context length and batch size, and at long contexts it rivals the weights themselves in size.

No cache:   step t → recompute K,V for tokens 1..t  (O(t) per step, O(n²) total)
With cache: step t → K,V for token t only + read
            cached K,V of tokens 1..t-1 from HBM    (O(1) compute, O(n) total)

⚠️ Common mistake: saying "Q, K and V are cached." Q is not cached — a token's query is used exactly once, at the moment it is generated.

Short answer: ≈ 0.5 MB per token → at 4k context ≈ 2 GB per sequence. With batch 16 that's ≈ 32 GB — more than the weights themselves (14 GB in fp16). This is THE canonical LLM-infrastructure calculation.

In depth:

Formula: 2 (K and V) × n_layers × n_kv_heads × head_dim × bytes/element.

Llama-2-7B, fp16:
  2 × 32 (layers) × 32 (KV heads) × 128 (head_dim) × 2 B
  = 524,288 B ≈ 0.5 MB per token

Context 4096:  0.5 MB × 4096 ≈ 2 GB per sequence
Batch 16:      2 GB × 16    ≈ 32 GB   (weights are only 14 GB)

1. Takeaway #1 — at long contexts memory is eaten by the cache, not the weights: this is exactly why GQA, PagedAttention and KV-cache quantization exist.
2. Takeaway #2 — the maximum batch, and therefore serving throughput, is bounded by KV memory, not compute.

⚠️ Common mistake: dropping the factor of 2 (K and V), or computing per batch when asked per sequence.

Short answer: Prefill processes the whole prompt in parallel — big matmuls, high arithmetic intensity → bounded by FLOPs. Decode emits one token per step, but each step streams ALL weights and the entire KV-cache from HBM → bounded by memory bandwidth, while the compute units idle.

In depth:

1. Key fact — decode must read every model weight for every token: speed ≈ bandwidth ÷ model size in bytes.
2. Consequence — almost every inference optimization targets exactly one of the two phases; this asymmetry is the skeleton of the whole topic.

⚠️ Common mistake: estimating decode speed from GPU FLOPs — it is predicted by memory bandwidth, not compute.

Short answer: TTFT (time to first token) — from request to first token: queueing + prefill. TPOT (time per output token) — the gap between subsequent tokens: decode speed. Total latency ≈ TTFT + TPOT × number of output tokens, so long outputs are almost entirely TPOT-dominated.

In depth:

1. Chat — the user feels TTFT ("is the model stuck?") and TPOT ("typing speed").
2. Long generation — with 1000 output tokens TTFT is barely noticeable: optimize TPOT.
3. Different knobs — speeding up prefill won't speed up decode and vice versa; measure which metric hurts first.

⚠️ Common mistake: optimizing "latency" in general without decomposing it into TTFT and TPOT — and investing in the wrong phase.

Short answer: KV-cache memory fragmentation. The classic approach gives each sequence a contiguous buffer sized to max_seq_len — 60–80% of memory is wasted. PagedAttention slices the cache into fixed-size blocks addressed through a block table, like virtual memory in an OS: almost all memory becomes useful, and the effective batch size and throughput jump.

In depth:

1. The problem — output length is unknown in advance: reserve for the maximum → internal fragmentation; allocate contiguously as you go → external fragmentation.
2. The solution — KV blocks of N tokens each, physically scattered across HBM; a block table assembles the logical sequence (a direct analogy to pages and a page table).
3. Bonus — a shared prefix (system prompt, few-shot) is stored once: blocks are shared between sequences with copy-on-write.

logical seq:    [block 0][block 1][block 2] …
                     │        │       │       block table
physical HBM:      #17       #4      #52      (any free blocks)

⚠️ Common mistake: describing vLLM as just "a fast engine." The core win is memory: a larger effective batch → higher throughput; attention itself doesn't get faster.

Short answer: A static batch is admitted as a whole and lives until the end: finished sequences wait for the longest one (head-of-line blocking). Continuous batching re-plans the batch on every iteration: completed sequences are immediately swapped for queued ones → 2–10× throughput, GPU utilization from ~30–40% to 80–90%.

In depth:

1. Root of the problem — autoregressive output lengths vary wildly and are unknown in advance: one answer is 10 tokens, its neighbor is 800.
2. Static — slots of finished sequences sit idle, the GPU grinds padding, new requests wait outside.
3. Continuous — an iteration-level scheduler (Orca, vLLM): every decode step is potentially a new batch composition.

Static:        A ████░░░░░░ (waits)
               B ██████████
               C ██░░░░░░░░ (waits)    → new requests wait outside

Continuous:    A ████ D ██████
               B ██████████
               C ██ E ████ F ███      → slot freed — instantly refilled

⚠️ Common mistake: explaining the win as "just a bigger batch." The essence is per-iteration swapping; without it, a batch of any size suffers head-of-line blocking.

Short answer: Decode is bounded by reading weights from HBM; int4 shrinks them 4× → tokens get almost that much faster, even though compute still runs in fp16 after dequantization. The price is typically ~1–2% on benchmarks, worse on math, code and long-tail domains.

In depth:

1. Mechanics — weights are stored in int4, dequantized to fp16 on the fly, then multiplied: you save memory traffic, not FLOPs.
2. GPTQ — layer-wise quantization that minimizes the layer's output error (Hessian-based approximation) on calibration data.
3. AWQ — uses activation statistics to find the ~1% salient weight channels and protects them via scaling, with no backpropagation.
4. Quality — degradation is uneven: average benchmarks barely move, while the long tail (math, code, rare knowledge) suffers first.

⚠️ Common mistake: expecting the same speedup on prefill — it is compute-bound, and saving bytes gains almost nothing there.

Short answer: Because of outliers: certain activation channels carry values orders of magnitude larger than the rest. They stretch the int8 dynamic range so far that "normal" values collapse into a handful of levels and accuracy falls apart. Weights, by contrast, are compactly distributed and quantize easily.

In depth:

1. Nature of the outliers — in large models (roughly 6–7B and up), channels with magnitudes tens of times above the median appear systematically; it is a property of trained transformers, not noise.
2. LLM.int8() — mixed precision: outlier channels are decomposed into a separate fp16 path, everything else is multiplied in int8.
3. SmoothQuant — migrates the difficulty from activations to weights: activations are divided by a per-channel scale s, weights are multiplied by s — mathematically equivalent, but both tensors become int8-friendly.

activations by channel:  ▁▁▁▂▁█▁▁▂▁▁█▁   ← 2 outlier channels set the whole range
after SmoothQuant:       ▂▂▂▃▂▄▂▂▃▂▂▄▂   ← range evened out, int8 suffices

⚠️ Common mistake: "int8 works the same everywhere." Weight-only int8/int4 is nearly free; activations break without outlier handling — and precisely on large models.

Short answer: A small draft model proposes k tokens; the target model scores all of them in one parallel forward pass. A token is accepted with probability min(1, p_target/p_draft); at the first rejection, a new token is resampled from the normalized (p_target − p_draft)₊. This is rejection sampling: the resulting distribution equals the target model's exactly — the method is lossless.

In depth:

1. Why it's faster — verifying k tokens is a single forward pass with prefill-like parallelism; there are fewer expensive target-model decode steps.
2. Acceptance rule — if p_draft ≤ p_target, the token is always accepted; otherwise with probability p_target/p_draft.
3. Correction on rejection — sample from max(0, p_target − p_draft), renormalized; this exact step completes the math to a strict equality of distributions.
4. Speedup ∝ acceptance rate — the closer the draft is to the target in domain and style, the longer the accepted runs; typically 2–3×.

draft:      t1 t2 t3 t4 t5   → target: one forward pass over all five
acceptance: ✓  ✓  ✗          → t1,t2 accepted, t3 resampled,
                               then the loop restarts

⚠️ Common mistake: calling the method "approximate" or "lossy." It yields exactly the same distribution — you sacrifice only extra compute, never quality.

Short answer: Quantize first: no training, done in hours, ~4× memory and noticeably faster decode. If that's not enough — distill: it needs a training budget but gives the best quality-per-FLOP and a smaller architecture outright. Pruning — structured only: unstructured sparsity rarely speeds up real GPUs.

In depth:

1. Ordering — cheap and reversible before expensive: quantize → (if needed) distill → pruning as a niche tool.
2. Methods compose — you quantize the distilled model too: the size win and the bit-width win multiply.

⚠️ Common mistake: proposing unstructured pruning for speed — 50% zeros without hardware sparsity support buy you almost nothing.

Short answer: They share K/V heads across groups of query heads: the KV-cache shrinks by n_heads/n_kv_heads. Llama-2-70B has 64 Q heads and 8 KV heads → an 8× smaller cache → longer contexts and bigger batches in the same memory. The price is a slight quality cost; GQA is usually trained in from the start.

In depth:

1. Link to the cache formula — in 2 × layers × n_kv_heads × head_dim × bytes, GQA reduces exactly the n_kv_heads factor.
2. Why it's a decode story — less cache is read from HBM per step → lower TPOT and more concurrent sequences.
3. When to apply — when training from scratch; converting an existing MHA model (uptraining) is possible but requires fine-tuning.

⚠️ Common mistake: "GQA speeds up training." The main win is inference memory and bandwidth; attention FLOPs barely change.

Short answer: Weights: 13B × 2 B ≈ 26 GB → ~50 GB left for the KV-cache. For 13B the cache is ≈ 1 MB/token → 2k tokens ≈ 2 GB per sequence → ~25 concurrent sequences. Then: total tokens/s and GPU price per hour → $/1M tokens.

In depth:

Memory:   80 − 26 (weights) − ~4 (activations, buffers) ≈ 50 GB for KV
KV:       13B ≈ 1 MB/token → 2048 tokens ≈ 2 GB/sequence
Batch:    50 / 2 ≈ 25 concurrent sequences

Speed:    decode is memory-bound → ~2 TB/s HBM ÷ 26 GB of weights
          ≈ 75 forward passes/s × batch 25 ≈ ~1500–1900 tok/s (rough)

Cost:     GPU $2/hr → 2 ÷ (1700 × 3600) × 10⁶ ≈ $0.3–0.4 per 1M tokens

1. Answer skeleton — memory → batch → throughput → $/token; the interviewer grades the structure of the Fermi estimate, not the third decimal.
2. Knobs — int4 weights free ~20 GB for cache and double weight-read speed; GQA cuts the 2 GB/sequence severalfold.

⚠️ Common mistake: counting only the weights and declaring "plenty fits" — in real serving it is the KV-cache that eats the memory.