10 DevOps Networking in Ops Interview Questions and Answers

Short answer: The app calls getaddrinfo → libc consults nsswitch.conf (/etc/hosts first) → the local stub resolver (systemd-resolved with its own cache) → a recursive resolver (ISP/CoreDNS) caching by TTL → root → TLD → the zone's authoritative server.

In depth:

1. nsswitch.conf and /etc/hosts — before any DNS: the line hosts: files dns means /etc/hosts wins over the network.
2. Stub resolver — systemd-resolved (127.0.0.53) keeps its own cache; flush it with resolvectl flush-caches.
3. Recursive resolver — caches answers by TTL and — often forgotten — NXDOMAIN too (negative caching, TTL taken from the SOA).
4. Recursion — root → TLD (.com) → the zone's authoritative server.

Debugging hop by hop: dig example.com (system resolver), dig @8.8.8.8 (a specific resolver), dig +trace (full recursion from the root, bypassing every cache), getent hosts (the app's actual path, through nsswitch).

app → getaddrinfo → nsswitch.conf → /etc/hosts
                                  → stub (systemd-resolved, cache)
                                  → recursive resolver (caches by TTL)
                                  → root → TLD → authoritative

⚠️ Common mistake: debugging with dig alone and skipping nsswitch and /etc/hosts (dig talks to DNS directly, bypassing both) — and negative caching: "I deleted the record, but the NXDOMAIN is still living out its TTL."

Short answer: "Intermittently" is the DNS signature: one dead address in the A-record set, a cached negative answer, split-horizon (different resolvers return different answers), or Kubernetes search-domain expansion (ndots). The method: dig against every resolver in the chain and compare answers and TTLs.

In depth:

1. A dead address in the set — DNS returns several A records round-robin; if one backend is down, every Nth request fails.
2. Negative caching — a resolver cached NXDOMAIN/SERVFAIL and serves it until the TTL expires, even though the record now exists.
3. Split-horizon — internal and external resolvers see different zones; which one your machine got is a resolv.conf question.
4. ndots in Kubernetes — a name with fewer than 5 dots is first expanded through the search domains (svc.cluster.local etc.): extra queries, timeouts, occasionally surprising name matches.

dig +short api.example.com                # system resolver
dig +short api.example.com @10.0.0.2      # each resolver from resolv.conf
dig +trace api.example.com                # what the authoritative serves
dig api.example.com                       # compare the ANSWER set and TTLs

⚠️ Common mistake: concluding "IP works, so the network is fine" and going off to debug the application. An intermittent failure by name is round-robin over a broken set or a cache race — not a code bug.

Short answer: The client sends ClientHello (versions, ciphers, and the hostname in SNI) → the server replies with its certificate and key-exchange parameters → both sides derive session keys, and traffic is then encrypted symmetrically. SNI exists because TLS happens BEFORE HTTP: the Host header hasn't been sent yet, so the server has nothing else to pick a certificate by.

In depth:

1. ClientHello — TLS versions, cipher list, SNI in plaintext; in TLS 1.3 the key share comes along too.
2. ServerHello + certificate — the server picks a cipher and the certificate matching the SNI.
3. Key exchange — (EC)DHE provides forward secrecy; session keys are derived from the shared secret.
4. Why SNI specifically — the hostname has to travel in the handshake: HTTP's Host header arrives inside the already-encrypted channel. That's how dozens of virtual hosts with different certificates share one IP:443.

⚠️ Common mistake: the expired certificate is the classic outage. "We'll renew by hand" is the wrong answer: ACME/cert-manager for auto-renewal plus an alert 2–3 weeks before expiry.

Short answer: An L4 balancer sees only IPs and ports — it spreads TCP/UDP flows, fast and cheap, but the protocol is opaque to it. L7 parses HTTP: routing by path and headers, retries, TLS termination, cookie affinity — at the cost of CPU spent parsing.

In depth:

Examples: L4 — IPVS, AWS NLB; L7 — nginx, Envoy, AWS ALB. A common cascade: L4 on the edge → L7 inside.

⚠️ Common mistake: expecting retries or path-based routing from L4 — it doesn't see HTTP and fundamentally can't tell where one request ends and the next begins.

Short answer: conntrack is netfilter's connection-state table; NAT and the stateful firewall are built on it. Every NAT'd flow takes an entry; once the table is full, new packets are silently dropped. The fix: raise the limit, remove extra NAT hops, tame the DNS traffic.

In depth:

1. Why the table exists — for the reply packet to get the reverse translation, the kernel remembers every flow: addresses, ports, state, timeout.
2. UDP counts too — every DNS query creates an entry that lingers ~30 s after the answer. Microservices without a DNS cache spawn thousands of entries — the classic Kubernetes production incident.
3. Diagnosis — compare the counter to the limit and see what's eating the table:

sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
conntrack -L -p udp --dport 53 | wc -l            # how much DNS ate
sysctl -w net.netfilter.nf_conntrack_max=1048576  # the tactical fix

4. Strategically — fewer NAT hops, a local DNS cache (NodeLocal DNSCache), an eBPF dataplane (Cilium) instead of kube-proxy/iptables — flows bypass conntrack entirely.

⚠️ Common mistake: cranking the limit and closing the ticket. The limit is symptomatic: the cause is usually a DNS storm or a redundant NAT, and it will be back as traffic grows.

Short answer: A packet walks the chains in a fixed order: PREROUTING (nat table — DNAT happens here) → routing decision → INPUT or FORWARD (filter) → POSTROUTING (nat — SNAT/masquerade here). Reply packets don't traverse the rules again: conntrack applies the reverse translation automatically.

In depth:

1. DNAT before routing — the destination address must be rewritten before the kernel decides where the packet goes: to itself (INPUT) or in transit (FORWARD).
2. filter in INPUT/FORWARD — this is where ordinary firewall rules live.
3. SNAT/masquerade on the way out — in POSTROUTING, once the egress interface is known.
4. conntrack — the DNAT rule fires only on a flow's first packet; the rest and the reply traffic are translated from the conntrack entry.

packet ─► PREROUTING (nat: DNAT) ─► routing decision
                                       │ local?
                                       ├─ yes ─► INPUT (filter) ─► process
                                       └─ no ──► FORWARD (filter)
                                                └─► POSTROUTING (nat: SNAT) ─► out

Publishing a container port (-p 8080:80) is exactly a DNAT rule in PREROUTING plus masquerade for the return path.

⚠️ Common mistake: looking for DNAT in the filter table — then being surprised that tcpdump past the translation point shows already-rewritten addresses: NAT happened before where you're looking.

Short answer: The classic MTU black hole: the tunnel eats part of the frame (an overlay shrinks the effective MTU; VXLAN costs 50 bytes), and Path MTU Discovery doesn't work because a firewall drops ICMP "fragmentation needed". Small packets squeeze through; large ones vanish silently.

In depth:

1. Mechanics — the app sends packets sized for MTU 1500, the tunnel adds headers, the packet no longer fits; with DF set it gets dropped and an ICMP type 3 code 4 should be sent back — but ICMP is blocked, so the sender never learns.
2. Signature — the TCP handshake and small responses are fine (small segments), data transfer hangs. Exactly "curl works, upload hangs".
3. Test — binary-search the size with fragmentation forbidden:

ping -M do -s 1472 host   # 1472 + 28 = 1500: does it pass?
ping -M do -s 1422 host   # narrow down to the real path MTU

4. Fix — MSS clamping at the tunnel boundary (--clamp-mss-to-pmtu), or lower the interface MTU, or allow the needed ICMP through the firewall.

⚠️ Common mistake: failing to connect "depends on size" with MTU and going off to debug the application. A size-dependent failure through a tunnel is MTU until proven otherwise.

Short answer: By itself — no: TIME_WAIT is the normal state of whichever side closed the connection first (it waits 2×MSL, usually 60 s, so stray late segments don't land in a new connection). It only hurts on the CLIENT side under high connection churn: ephemeral ports run out.

In depth:

1. Why the state exists — protection from wandering segments of the old connection and reliable teardown: the final ACK can get lost.
2. When it hurts — a client (a proxy, a service opening a connection per request) piles up tens of thousands of TIME_WAITs and exhausts the ~28k ephemeral ports per (dst IP, dst port) tuple — new connects fail with EADDRNOTAVAIL.
3. The proper fix — keep-alive and connection pooling: stop opening a connection per request.
4. Knobs — tcp_tw_reuse is safe for outbound connections (reuse guarded by TCP timestamps); widen ip_local_port_range.

ss -s                                    # summary: how many timewait
ss -tan state time-wait | awk '{print $4}' | sort | uniq -c | sort -rn | head

⚠️ Common mistake: recommending tcp_tw_recycle — it broke clients behind NAT (timestamps from different machines interleave) and was removed from the kernel as of 4.12.

Short answer: The health check probes something other than what traffic lives on: /healthz responds while /api dies on a dependency. The other classic is a keep-alive idle-timeout mismatch: the balancer holds the connection longer than the app, reuses an already-closed socket, and gets a 502.

In depth:

1. Probe ≠ real path — a dependency-free check stays green while the real request dies on the database or an external API. You need a readiness probe that touches real dependencies (carefully: don't build a cascade).
2. The keep-alive race — the app closes an idle connection after 5 s, the balancer considers it alive for 60 s: the next request rides into a closed socket → 502. The rule: the app's idle timeout must exceed the balancer's.
3. 502 vs 504 — different diseases: 502 — the backend answered "wrongly"; 504 — it didn't answer at all.

⚠️ Common mistake: trusting a green health check. It answers "is the process alive", not "is it serving real requests".

Short answer: Each version removes its own layer of head-of-line blocking (HOL). In HTTP/1.1 requests on a connection are strictly sequential; HTTP/2 multiplexes streams over one TCP connection, but one lost packet stalls all streams (TCP-level HOL); HTTP/3 (QUIC over UDP) gives independent streams and 0-RTT.

In depth:

1. HOL in 1.1 — a slow response holds everyone behind it; pipelining is broken in practice.
2. TCP HOL in h2 — TCP guarantees byte order: a hole in the stream blocks delivery of all streams, even when their bytes have already arrived.
3. QUIC — reliability per stream, with the handshake merged into TLS 1.3.

⚠️ Common mistake: "h2 solved HOL completely." It removed HOL at the HTTP level, but on a lossy network h2 over a single TCP connection can lose even to 1.1 with six connections.