12 NLP and LLMs Interview Questions and Answers

Short answer: An embedding is a dense vector encoding a token's meaning so that semantically similar items sit close in vector space. The key shift: word2vec gives a static vector — one per word, always — while BERT-style models give a contextual one: the vector depends on the sentence, finally solving polysemy.

In depth:

1. word2vec (skip-gram) — learns to predict context words from a word; after training it is a lookup table "word → vector". "Bank" of a river and "bank" with your money get the same vector.
2. Contextual embeddings — a word's vector is computed by a transformer over the whole sentence; "bank" gets different vectors in different phrases.
3. Pretraining objectives differ — skip-gram/CBOW for word2vec vs masked LM for BERT: predict a masked token from bidirectional context.

⚠️ Common mistake: reciting "king − man + woman ≈ queen" but failing to name the real win of contextual embeddings — solving polysemy.

Short answer: Attention(Q,K,V) = softmax(QKᵀ/√d)·V. Each token forms a query Q ("what I'm looking for"), a key K ("what I contain") and a value V ("what I give away"). A token's new representation is a weighted sum of all tokens' V, with weights = softmax over the similarity of its Q to their K. Dividing by √d keeps dot products from growing with dimension and saturating the softmax.

In depth:

1. Q, K, V — three linear projections of the same input: X·W_Q, X·W_K, X·W_V.
2. QKᵀ — the all-pairs similarity matrix: how relevant token j is to token i.
3. √d — the variance of q·k grows like d; without scaling, softmax collapses to near one-hot, gradients vanish, training stalls.
4. Multi-head — h independent lower-dimensional attentions capture different relation types (syntax, coreference) in parallel.

"the cat sat on the mat" — computing token "sat":
score = Q(sat)·K(x):  cat 9.1 | on 2.0 | mat 7.3
softmax(score/√d)  →  0.60    | 0.05   | 0.35
new(sat) = 0.60·V(cat) + 0.05·V(on) + 0.35·V(mat)

⚠️ Common mistake: stating the formula but not explaining √d. "Why √d exactly?" is the standard follow-up: a dot product of d independent components has variance ~d, hence normalize by √d.

Short answer: Self-attention is permutation-invariant: without positions, "dog bites man" and "man bites dog" produce identical representations. Positional encodings inject token order; the main variants are sinusoidal, learned, RoPE and ALiBi.

In depth:

1. The reason — attention is a weighted sum over a set of tokens, and a sum doesn't know the order of its terms. Order must be added explicitly.
2. The variants:

3. RoPE and context length — its relative nature extrapolates better beyond the trained length; context extensions (position interpolation, YaRN) are built around RoPE.

⚠️ Common mistake: listing the types without answering "why". The core of the answer is attention's permutation invariance; without it the interviewer hears memorization.

Short answer: Three reasons: parallel training (an RNN processes tokens strictly sequentially), an O(1) path between any two tokens versus O(n) for recurrence, and predictable scaling with data and compute. The price: attention costs O(n²) in sequence length.

In depth:

1. Parallelism — an RNN must process token t−1 before token t; a transformer computes attention for all pairs in one matrix multiplication → GPUs stay busy, training is orders of magnitude faster.
2. Long-range dependencies — in an RNN, signal between distant tokens travels n steps and decays; in attention any two tokens connect directly.
3. LSTM in context — the gates (input/forget/output) solved vanishing gradients but not the sequential nature of the computation.
4. Scaling — transformers improve predictably with data and compute (scaling laws) — the foundation of modern LLMs.

⚠️ Common mistake: omitting the O(n²) caveat. "Transformers are better at everything" signals not knowing the price — quadratic attention is why the whole long-context optimization industry exists.

Short answer: BERT is a bidirectional encoder pretrained with masked LM: it sees context on both sides and excels at understanding — classification, NER, embeddings. GPT is a causal decoder pretrained on next-token prediction: the attention mask forbids looking ahead, which is exactly what lets it generate text.

In depth:

1. Attention direction — encoder: every token sees the whole sentence; decoder: only the past (causal mask), otherwise predicting the next token would be cheating.
2. Objectives — masked LM: reconstruct ~15% masked tokens from bidirectional context; causal LM: predict the next token.
3. Consequence for tasks — understanding vs generation: embeddings for search and retrieval are usually encoder-style (E5, BGE); dialogue and generation are decoders.

⚠️ Common mistake: "GPT is just newer and better." For classification and retrieval at the same budget, a compact encoder still often wins — the question is about matching architecture to task.

Short answer: In training and prefill — exact IO-aware optimizations (FlashAttention) and sparse schemes (sliding window); at generation — the KV-cache: keys and values of past tokens are computed once and stored, so a generation step costs O(n) instead of recomputing everything. The KV-cache is exactly why long contexts eat memory at inference.

In depth:

1. FlashAttention — exact attention reordered for the GPU memory hierarchy: tiling, no materialization of the n×n matrix. Same numbers, several times faster and leaner on memory.
2. Sparse / sliding window — each token attends to a window of w neighbors: O(n·w) instead of O(n²).
3. Linear approximations — Performer, linear attention: approximate the softmax, but barely caught on in frontier models.
4. KV-cache — during autoregressive generation past K and V never change; cache them and compute attention only for the new token.
5. MQA/GQA — fewer KV heads → a several-times smaller cache at nearly the same quality.

Generation step t:
no cache:  recompute K,V for all t tokens → O(t²) per step
KV-cache:  past K,V already in memory     → O(t) per step
cache size ≈ 2 · layers · KV-heads · d · t → grows with context

⚠️ Common mistake: never having heard of the KV-cache. "Why is long context expensive in memory specifically at inference?" catches exactly this.

Short answer: An escalation ladder. Prompting first — cheapest, iterate in minutes. Then RAG — when you need private, fresh, or attributable knowledge: grounding, citations, an updateable index with no training. Only then fine-tuning (usually LoRA/PEFT) — when you need stable behavior: style, format, domain terminology. Fine-tuning is not a way to "load facts" into a model.

In depth:

1. The key axis — knowledge vs behavior: knowledge lives in the index (RAG), behavior lives in the weights (fine-tuning).
2. Combinations are legal — RAG for facts + LoRA for tone and answer format is a standard production setup.

⚠️ Common mistake: "let's fine-tune the model on our docs so it knows them" — the canonical wrong answer of 2025–2026 screens. For knowledge, use RAG; teaching facts via fine-tuning is unreliable and goes stale instantly.

Short answer: LoRA freezes the pretrained matrix W and learns a low-rank update ΔW = B·A with rank r ≪ d on the attention/MLP projections. Only ~0.1–1% of parameters train, it fits on a single GPU, and after training the adapter merges into the weights — zero added inference latency. It works because task adaptation lives in a low-rank subspace: you don't need full rank to shift behavior.

In depth:

1. Mechanics — h = W·x + B·A·x; A is initialized randomly, B with zeros, so training starts at ΔW = 0 and the base model's behavior isn't broken.
2. The savings — at d = 4096 and r = 16: 16.7M parameters in the matrix vs 131K in the adapter (~0.8%).
3. Merging — W' = W + B·A is computed once before inference: no extra multiplication in production; adapters can be stored per task.
4. QLoRA — base model in 4-bit + LoRA on top: fine-tune a 70B on one card.

h = W·x + B·(A·x)
W: d×d — frozen
A: r×d, B: d×r, r ≪ d — trained
d=4096, r=16: 16.7M → 131K parameters (0.8%)

⚠️ Common mistake: being unable to say what exactly is frozen and what is trained. "LoRA means training fewer layers" is wrong: you train low-rank additions to matrices, not a subset of layers.

Short answer: Pretraining teaches exactly one thing — continuing text, not following instructions. Asked "how do I bake bread?", a base model may answer with a list of similar questions: that's a perfectly good text continuation. SFT on instruction → response pairs teaches the dialogue format; RLHF/DPO optimizes a reward built from human preferences — helpful, harmless, honest.

In depth:

1. Pretraining — next token prediction over trillions of tokens: the knowledge and capabilities are there, the "assistant" interface is not.
2. SFT — supervised fine-tuning on instruction pairs: the model learns the role and the response format.
3. RLHF — a reward model trained on human comparisons of answer pairs + RL (PPO) against it; DPO reaches the same goal directly, without an explicit reward model.
4. Alignment tax — alignment can slightly degrade raw capabilities (benchmarks); a deliberate trade-off in favor of steerability.

Pretraining (the whole internet)  → "continue the text"
   ↓ SFT (instruction → response) → "answer in assistant format"
   ↓ RLHF / DPO (preferences)     → "helpful, harmless, honest"

⚠️ Common mistake: conflating knowledge with behavior — assuming instruction-following "emerges by itself" from pretraining. The interviewer is listening for whether you separate these two layers.

Short answer: They are ways to pick the next token from the predicted distribution. Greedy takes the argmax and tends to loop; beam searches for a globally likely sequence (translation, summarization); top-k and top-p truncate the tail of the distribution before sampling; temperature divides the logits, making the softmax sharper (T < 1) or flatter (T > 1).

In depth:

1. Why top-p is adaptive — in a confident distribution the cut is narrow (2–3 tokens), in an uncertain one it's wide; top-k cuts the same width always.
2. Combinations — in practice temperature + top-p are used together.

⚠️ Common mistake: saying "temperature makes the model more creative" without the mechanics. The strong answer: T rescales logits before the softmax, redistributing mass between the head and the tail.

Short answer: The training objective is next-token plausibility, not truth: the model is trained to generate text that looks right. Add no grounding in sources, and knowledge frozen at the data cutoff. Hallucinations can be reduced — never fully eliminated.

In depth:

1. RAG with citations — anchor the answer to retrieved documents: the model answers from context and cites sources, so errors become checkable.
2. Structured output — constrained decoding, JSON schemas: fewer degrees of freedom to invent things in critical fields.
3. Lower temperature — on factual tasks, less randomness from the tail of the distribution.
4. Self-consistency and verification — several samples with answer cross-checking; a separate pass that verifies claims against sources.
5. Evals and human-in-the-loop — regression suites for factuality; for high stakes (medicine, law, money) mandatory human review.
6. An honest "I don't know" — via prompting and training, reward refusal over confident invention.

⚠️ Common mistake: claiming that something — RAG, fine-tuning, a "don't make things up" prompt — eliminates hallucinations. Every measure reduces the rate; "we got it to zero" is a red flag for the interviewer.

Short answer: Chunking → embeddings → vector index (FAISS/HNSW) → retrieve top-k (hybrid BM25 + dense) → reranking with a cross-encoder → prompt with citations → generation. It breaks at every link: bad chunking, retrieval misses, lost-in-the-middle, a stale index. Evaluate the retriever and the generator separately.

In depth:

Documents → chunking → embeddings → index (FAISS/HNSW)
Query ──→ retrieve top-k (BM25 + dense) → rerank (cross-encoder)
       └→ prompt with citations → LLM → answer + sources

1. Failure points — a chunk cuts a thought in half; the needed document isn't in the top-k; the relevant chunk gets lost in the middle of a long context (lost-in-the-middle); the index isn't rebuilt after document updates.
2. Hybrid + rerank — BM25 catches exact terms and abbreviations, dense catches paraphrases; a cross-encoder is more accurate than a bi-encoder but expensive, so only over the top-k.
3. Evaluating the retriever — recall@k, MRR on labeled "query → correct chunk" pairs, separately from generation.
4. Evaluating generation — faithfulness (no claims beyond the context) and relevance; LLM-as-judge only with a rubric validated against human labels.

⚠️ Common mistake: telling only the happy path. The interviewer expects failure points and separate retrieval vs generation evaluation — otherwise you can't tell which link to fix.