12 Deep Learning Interview Questions and Answers

Short answer: Backpropagation is the chain rule applied to the computational graph: the forward pass caches activations, the backward pass propagates the loss gradient from the output down to every weight. Zero initialization breaks training because of symmetry: every neuron in a layer receives identical gradients and stays an exact copy of its neighbors forever.

In depth:

1. Forward pass — compute the output and the loss while caching intermediate activations: they are needed for the derivatives.
2. Backward pass — walk the graph in reverse; each node's gradient = local derivative × upstream gradient (chain rule).
3. Update — one gradient descent step on every weight.
4. Symmetry with zeros — if all weights are equal, every neuron in a layer produces the same output and the same gradient → after any number of steps they remain identical; the network collapses to one neuron per layer.
5. The right way — random initialization: He for ReLU, Xavier for tanh/sigmoid.

x ──► h = f(W1·x) ──► ŷ = g(W2·h) ──► L
∂L/∂W1 = ∂L/∂ŷ · ∂ŷ/∂h · ∂h/∂W1   ← chain rule

⚠️ Common mistake: being unable to say “chain rule”, or to explain that it is symmetry — not “zero gradients” — that kills zero init: the layer's gradients are not zero, they are identical.

Short answer: The gradient reaching early layers is a product of many Jacobians. If the factors are systematically below one (sigmoid/tanh saturation, small weights), the gradient decays exponentially; above one — it explodes. The standard toolkit: the ReLU family, He/Xavier initialization, BatchNorm/LayerNorm, residual connections, gradient clipping.

In depth:

1. Mechanics — by the chain rule, layer k's gradient contains the product of every Jacobian above it: the effect is exponential in depth.
2. Saturation — the sigmoid derivative is ≤ 0.25 and tanh has flat tails too: a deep stack of them almost surely vanishes.
3. RNNs are a special pain — the same matrix is multiplied T times along the sequence, so explosions are especially typical there; clipping the gradient norm is standard.

⚠️ Common mistake: answering with the single bullet “use ReLU”. The interviewer expects a system: activations + initialization + normalization + skip connections — different levers against the same exponential.

Short answer: During training, dropout zeroes random activations with probability p: every batch trains its own “thinned” subnetwork, which prevents neuron co-adaptation and acts like an ensemble of subnetworks. At inference dropout is OFF; to keep activation scale consistent, inverted dropout divides by (1−p) already at training time.

In depth:

1. Mechanics — a binary mask over activations, resampled every forward pass; a neuron cannot rely on specific neighbors and learns more robust features.
2. Interpretation — training an exponential number of weight-sharing subnetworks; inference ≈ averaging that ensemble.
3. Inverted dropout — activations are divided by (1−p) during training, so nothing needs rescaling at inference — the layer simply switches off.
4. Where to put it — usually after fully-connected layers; in convnets it is more often replaced by BatchNorm and augmentations.

drop = nn.Dropout(p=0.3)
model.train()   # mask active, the (1-p) rescaling is built in
model.eval()    # dropout off: deterministic inference

⚠️ Common mistake: forgetting the train/inference difference (and model.eval() in PyTorch): dropout left on at inference produces noisy predictions.

Short answer: During training, BatchNorm normalizes pre-activations using the current batch statistics and applies a learnable scale and shift γ/β, updating running averages along the way. At inference there may be no batch — the accumulated running mean/var are used, and the layer becomes a deterministic affine transform. It helps because it smooths the loss landscape and enables higher learning rates.

In depth:

1. Train — x̂ = (x − μ_batch)/√(σ²_batch + ε), then y = γ·x̂ + β; γ and β are learned by gradient descent, running statistics are updated with an exponential moving average.
2. Inference — the same formulas but with running mean/var: the prediction does not depend on batch neighbors.
3. Why it works — the canonical “internal covariate shift” story is contested (Santurkar et al., 2018): the working explanation is smoothing of the optimization landscape → robustness to large LRs and to initialization.
4. Bonus — mild regularization: the noise of batch statistics acts like weak dropout.

Classic follow-up: which layers behave differently in train and eval? — BatchNorm and dropout; hence the mandatory model.train()/model.eval().

⚠️ Common mistake: confidently selling “eliminating internal covariate shift” as fact. A strong answer notes the explanation is disputed, while the empirics — larger LRs and faster convergence — are not.

Short answer: BatchNorm normalizes each channel across the batch — its statistics depend on batch neighbors. LayerNorm normalizes the features within a single sample — independent of both batch size and sequence length. Hence BN is the standard in CNNs, and LN in transformers and RNNs.

In depth:

1. Normalization axes — BN: averaging over (N, H, W) separately per channel; LN: averaging over the features of one sample.
2. When BN breaks — tiny batches (noisy statistics), variable-length sequences, train/eval mismatch through running statistics.
3. Why LN in transformers — works identically for a batch of 1 and of 1024, carries no state (running stats), and plays well with token-by-token autoregressive inference.

⚠️ Common mistake: “LN is just BN rotated 90°, no real difference.” The difference is operational: BN carries state and couples samples within a batch, LN is a pure function of a single sample.

Short answer: SGD steps along the stochastic gradient. Momentum adds velocity accumulation (an exponential average of gradients) — damping oscillations and accelerating movement along the consistent direction. Adam adds, on top of momentum, a per-parameter adaptive learning rate from second-moment estimates of the gradients — which is why it is the default in practice.

In depth:

1. SGD — w ← w − lr·g; noisy, sensitive to the lr choice and feature scale.
2. Momentum — v ← β·v + g, step along v: oscillations across the loss “ravine” cancel out, movement along it accumulates.
3. Adam — first moment m (direction) + second moment v (scale) → step m̂/√v̂: parameters with rare or small gradients get a larger effective lr.
4. Practice — Adam/AdamW as the default; SGD+momentum sometimes generalizes better on vision tasks given a good lr schedule.

Bonus for 🔴 level: in AdamW, weight decay is decoupled from the adaptive step — an L2 penalty inside Adam is not equivalent to honest decay, which is why AdamW became the standard.

⚠️ Common mistake: reciting optimizer names as magic words without being able to say which problem each addition solves.

Short answer: The sigmoid derivative never exceeds 0.25 and is nearly zero in the tails — a deep stack of sigmoids drowns in vanishing gradients. ReLU has derivative exactly 1 for x > 0: the gradient passes through without saturation; on top of that ReLU is dirt cheap to compute, and half-zero activations give sparsity.

In depth:

1. Gradients — the main argument — no saturation on the positive half-axis → the signal reaches early layers even in a deep network.
2. Cheap — max(0, x) versus computing an exponential.
3. Sparsity — only a fraction of neurons fire on each example.
4. The catch: dying ReLU — a neuron pushed into the negative zone has zero gradient and can “die” permanently → LeakyReLU or GELU (the transformer standard).

⚠️ Common mistake: answering “it's faster” or “it's the convention” without mentioning gradient saturation — which is the core of the question.

Short answer: Convolution bakes in two image priors: locality (a pixel is related to its neighbors) and weight sharing (one filter scans the whole image). This yields translation equivariance and orders of magnitude fewer parameters than a fully-connected layer. The receptive field is the input region one neuron “sees”; it grows with network depth.

In depth:

1. Parameters — a fully-connected layer from 224×224×3 to 1000 neurons ≈ 150M weights; a 3×3 convolution from 3 to 64 channels — under 2 thousand.
2. Translation equivariance — the object shifts, the feature map shifts with it; a fully-connected net would have to learn every shift separately.
3. Receptive field — grows with every layer: a stack of small convolutions covers the same area as one large filter, but cheaper and with extra nonlinearities.
4. Interview mini-task — the conv output size: (n + 2p − k)/s + 1; interviewers ask to compute it by hand.

Input 32×32, conv k=5, p=2, s=1:  (32 + 4 − 5)/1 + 1 = 32
Input 32×32, conv k=3, p=0, s=2:  (32 − 3)/2 + 1 = 15
Receptive field: conv3×3 → conv3×3 = 5×5; one more layer = 7×7

⚠️ Common mistake: fumbling the output-size formula — interviews make you compute it by hand more often than you'd expect.

Short answer: The degradation problem: deep “plain” networks underperform shallow ones even on the training set — an optimization difficulty, not overfitting and not just vanishing gradients. The identity shortcut gives the gradient a direct highway back, and gives the layers an easier job: learn the correction F(x) = H(x) − x instead of the whole mapping.

In depth:

1. The symptom — a 56-layer plain net is worse than a 20-layer one on training error (He et al., 2015); overfitting cannot explain that.
2. The residual formulation — a block outputs y = F(x) + x; learning “do nothing” (F ≈ 0) is trivial, so added depth at worst does no harm.
3. The gradient highway — through the shortcut, the gradient flows straight to early layers, bypassing the product of Jacobians.
4. The consequence — networks of 100+ layers became trainable; the idea moved into transformers: residuals around attention and FFN blocks.

x ──► [conv → BN → ReLU → conv] ──► F(x) ──(+)──► ReLU ──►
 └────────────────── identity ──────────────┘

⚠️ Common mistake: reducing it all to “fighting vanishing gradients”. BatchNorm already keeps gradients alive in plain nets — the degradation is genuinely an optimization issue, and residuals fix it.

Short answer: Tabular data of modest size — boosting (XGBoost/LightGBM/CatBoost): on tables it still consistently beats neural nets on benchmarks, is cheaper to train, and easier to interpret. Deep learning is for unstructured data (text, images, audio), very large datasets, and wherever you need embeddings, transfer learning, or multi-task setups.

In depth:

1. The practical rule — even if a neural net is the plan, start with boosting as the baseline: it sets the bar and exposes data problems.
2. Hybrids — neural-net embeddings fed as features into boosting is a stock production pattern.

⚠️ Common mistake: “neural nets are always stronger.” The interviewer is listening for pragmatism: a choice driven by data type, volume, and budget — not hype.

Short answer: An embedding is a learned dense vector representing a discrete object (a word, product, user, category) such that the geometry of the space encodes semantic similarity: similar objects sit close together. It is both the way to feed categorical data to a neural net and a ready-made tool for similarity search.

In depth:

1. Versus one-hot — one-hot is sparse, vocabulary-sized, and every object in it is equidistant; an embedding is dense, low-dimensional, and its distances are meaningful.
2. Where it comes from — either learned jointly with the main task (nn.Embedding), or taken pretrained: word2vec, BERT layers, CLIP.
3. Why it matters in practice — candidate generation in recommenders (ANN nearest-neighbor search), semantic search, deduplication, features for boosting, transfer learning.

emb = nn.Embedding(num_embeddings=50_000, embedding_dim=64)
v = emb(token_ids)      # [batch, seq, 64] — dense vectors
# similarity = cosine similarity between vectors

⚠️ Common mistake: giving the definition without naming a single concrete use case. “Recommendation candidates via nearest-neighbor search over embeddings” is already a sufficient answer.

Short answer: The standard arsenal: quantization (fp16/int8), pruning, distillation (a compact student mimics a large teacher), hardware-targeted compilation (ONNX Runtime/TensorRT), request batching, and caching embeddings. And remember: inference is inherently lighter than training — it stores no activations or optimizer state, which is why training eats ~3–4× more memory.

In depth:

1. Order of operations — profile first: the bottleneck is often preprocessing or the network, not the model itself.
2. Memory — no backward pass at inference: no stored activations, gradients, or Adam moments.

⚠️ Common mistake: proposing “get a bigger GPU.” The interviewer expects the arsenal of techniques and the quality/speed trade-off of each.