13 DevOps Kubernetes Interview Questions and Answers

Short answer: kubectl validates the manifest and sends it to the API server: authn → RBAC → admission webhooks → persisted to etcd. That is where the synchronous part ends — controllers, the scheduler, and kubelet notice changes asynchronously via watch and drive the cluster toward the desired state. Nobody pushes anything.

In depth:

1. API server — authentication, authorization (RBAC), admission (mutating → validating webhooks), object stored in etcd.
2. Deployment controller — sees the new Deployment (watch) and creates a ReplicaSet.
3. ReplicaSet controller — creates Pod objects, with no node assigned yet.
4. Scheduler — finds pods with an empty nodeName, filters and scores nodes, binds.
5. kubelet — on the chosen node sees "its" pod: pulls images, starts containers via CRI, networking comes up via the CNI plugin.
6. Readiness — probe passes → the pod joins the EndpointSlice and starts receiving traffic.

kubectl ──► API server (authn → RBAC → admission) ──► etcd
              ▲ watch      ▲ watch       ▲ watch      ▲ watch
  deploy-ctrl → RS    rs-ctrl → Pods   scheduler → bind   kubelet → CRI/CNI

⚠️ Common mistake: "the API server schedules the pod and starts containers." It only stores and serves state — all the work is done by independent controllers via the watch/reconcile model.

Short answer: The apiserver is the single entry point and the only component that talks to etcd; etcd stores the state; the scheduler picks nodes; the controller-manager runs reconcile loops; kubelet runs containers on the node; kube-proxy programs the networking for Services.

In depth:

The first four are the control plane; kubelet and kube-proxy live on every node. The key idea: everything talks only to the apiserver via watch — components don't know about each other and never touch etcd directly. That's why the control plane survives the loss of any component except etcd.

⚠️ Common mistake: saying the scheduler or controller-manager "writes to etcd." They write to the apiserver — it alone holds the etcd connection.

Short answer: etcd runs Raft: a write commits once a majority — (n/2)+1 — acknowledges it. An even member adds no fault tolerance: both 3 and 4 nodes survive exactly one failure. When quorum is lost, etcd stops accepting writes → the API "freezes": kubectl apply fails, controllers and scheduling stall. Already-running pods keep running.

In depth:

1. Quorum lost — writes are impossible; the apiserver can still serve reads, but accepts no changes: no rollouts, no scaling, no new pods.
2. The data plane lives on — kubelet keeps containers running, kube-proxy keeps its rules; the cluster is frozen, but traffic flows.
3. A slow etcd is worse than a dead one — every API request gets slower, watches lag, controllers react late, leader election flaps.

⚠️ Common mistake: "etcd is down — everything is down." No: the management plane is down; workloads keep serving traffic until you need to change something.

Short answer: Pending means "the scheduler found no node." The first step is always the same: kubectl describe pod — in Events the scheduler explains exactly what it disliked. The cause is almost always one of five: resources, taints, affinity, PVC, quota.

In depth:

kubectl describe pod app-7d9f   # Events: "0/12 nodes are available: ..."
kubectl get nodes               # nodes alive and Ready?
kubectl describe node n1        # Allocatable vs Allocated: do the requests fit?
kubectl get pvc                 # is the PVC Pending too?
kubectl describe quota -n team  # hit the ResourceQuota?

1. Insufficient resources — the pod's requests don't fit on any node ("Insufficient cpu/memory").
2. Taints without tolerations — "node(s) had untolerated taint".
3. nodeSelector / affinity — no node matches the labels.
4. Unbound PVC — the pod waits for its volume: no StorageClass, zone mismatch.
5. ResourceQuota — the nuance: the pod isn't created at all; the error shows in the ReplicaSet's Events, not as Pending.

⚠️ Common mistake: starting with logs. A Pending pod has no logs — it hasn't started anywhere yet; everything you need is in Events.

Short answer: CrashLoopBackOff means the container starts, dies, and kubelet restarts it with a growing delay (10s, doubling, capped at 5m). The order: describe (Events and Last State with the exit code) → logs --previous (the logs of the container that actually crashed) → interpret the exit code.

In depth:

kubectl describe pod app-7d9f       # Events + Last State: Exit Code
kubectl logs app-7d9f --previous    # logs of the CRASHED container — the key flag
kubectl logs app-7d9f -c init-db    # if an init container is failing

Then, in descending frequency: broken config or secret (missing env, volume didn't mount), a failing liveness probe (Events will say "Liveness probe failed… will be restarted"), a dependency or migrations unavailable at startup.

⚠️ Common mistake: running kubectl logs without --previous and wondering why it's empty: the freshly restarted container hasn't written anything yet — the crashed one's logs live behind the flag.

Short answer: The Linux kernel kills it. Kubernetes merely wrote the memory limit into the cgroup (memory.max); when the process exceeds it, the kernel OOM killer fires and sends SIGKILL (128+9=137). The pod's QoS class sets oom_score_adj — deciding whom the kernel and kubelet sacrifice first.

In depth:

Two distinct mechanisms people conflate:

1. cgroup OOM (kernel) — the container exceeded ITS OWN limit → instant SIGKILL, status OOMKilled. Kubernetes is not in the loop at that moment.
2. Node-pressure eviction (kubelet) — the node as a whole runs out of memory → kubelet evicts pods itself, ranking by QoS and usage-over-request; status Evicted, not OOMKilled.

⚠️ Common mistake: "kubelet watches memory and kills the container when it hits the limit." No: the per-container limit is enforced by the kernel via cgroups; kubelet only steps in under whole-node pressure.

Short answer: Requests are the scheduling currency: the scheduler bin-packs pods onto nodes by them, and they set the CPU weight under contention. Limits are the ceiling: exceeding the CPU limit means throttling, exceeding the memory limit means SIGKILL (OOMKilled). CPU is a compressible resource, memory is not — hence the different punishment.

In depth:

1. Requests ≠ actual usage — the scheduler only sums requests: a node can be "full" on paper and idle in reality, and vice versa.
2. Limits without requests — Kubernetes automatically sets requests = limits.
3. Requests without limits — the pod can burst into free resources; often desirable for CPU, but for memory it risks an OOM on an oversubscribed node.

⚠️ Common mistake: "memory gets throttled above the limit, like CPU." Memory can't be taken away from a process gradually — only by killing it: exceeding the memory limit always ends in SIGKILL.

Short answer: A ClusterIP is a virtual address: nothing listens on it, and no packet ever "reaches" it. kube-proxy on every node programs iptables (or IPVS) rules: a packet to the ClusterIP gets DNAT'ed to the IP:port of one ready pod right in the node's netfilter. kube-proxy itself is not in the packet path.

In depth:

Pod A ──► 10.96.0.10:80 (ClusterIP: virtual, no process)
             │ iptables DNAT (rules written by kube-proxy)
             ▼ a random ready endpoint
          10.244.1.5:8080 (Pod B)

1. EndpointSlice — a controller maintains the list of the Service's pods that passed readiness; kube-proxy watches it and rewrites the rules.
2. Readiness = membership — a pod fails its probe → drops out of the EndpointSlice → no rule points at it anymore.
3. iptables vs IPVS — with thousands of Services the linear iptables chain gets slow; IPVS provides a hash table.
4. Ingress — that's L7: a separate controller (nginx, envoy) terminates HTTP and routes by host/path; the Service underneath is the same L4 machinery.

⚠️ Common mistake: thinking kube-proxy is an in-path proxy that packets flow through. It only configures kernel rules and never sees the traffic.

Short answer: Readiness controls traffic (EndpointSlice membership), liveness controls the container's life (failure = restart), startup delays the other two while the app boots. The classic outage: a liveness probe that checks a dependency (the DB) — the database blips, and kubelet restarts the entire fleet at once.

In depth:

Anatomy of the incident: liveness hits /health, which calls the DB → the database is down for 30 seconds → liveness fails on every replica → kubelet restarts all containers → the whole service is down, and the mass restarts finish off the barely recovered database.

The rule: liveness is about the process itself (deadlock, stuck event loop); dependencies are readiness territory — and even there, carefully.

⚠️ Common mistake: one endpoint for both liveness and readiness that checks everything. A single dependency blip becomes a fleet restart instead of a traffic pause.

Short answer: Deployment — interchangeable stateless replicas with rolling updates. StatefulSet — when replicas need stable identity: a name (pod-0, pod-1), a PVC per replica, ordered rollout, stable DNS via a headless Service. DaemonSet — exactly one pod on every matching node: log, metrics, CNI agents.

In depth:

A headless Service (clusterIP: None) gives every StatefulSet replica a predictable DNS name like web-0.web.ns.svc — that's how brokers and replicas find each other.

⚠️ Common mistake: "there's a database, so StatefulSet." If the DB is managed (RDS, Cloud SQL), the app is fine as a Deployment; and serious in-cluster databases usually run under an operator, which decides what to manage itself.

Short answer: By the formula: desiredReplicas = ceil(current metric / target × current replicas). For CPU, "utilization" is computed as a percentage of requests — no requests means no denominator, and the HPA can't compute anything at all.

In depth:

desired = ceil( current / target × currentReplicas )
example: CPU at 80% of request, target 50%, 4 replicas
         ceil(80/50 × 4) = ceil(6.4) = 7

1. Metric sources — metrics-server (resource metrics) or the custom/external metrics API (RPS, queue depth).
2. % of requests — "CPU 80%" means "the pod uses 80% of its request," not of the node; undersized requests make the HPA scale too early, oversized ones mean it never scales.
3. Stabilization — a 300s window on scale-down by default (scale-up is immediate): protection against flapping on a noisy metric.
4. Bounds — it never goes below minReplicas; when metrics are unavailable, scaling freezes rather than "scaling to zero."

⚠️ Common mistake: pointing an HPA at pods without requests and expecting magic — the status will show FailedGetResourceMetric and the replica count won't move.

Short answer: Because of a shutdown race: the SIGTERM to the container and the pod's removal from the EndpointSlice happen in parallel, while iptables on the nodes and external LBs update with a lag — for a while traffic keeps hitting an app that is already dying. The cure is a combo: a preStop pause + proper SIGTERM handling + headroom in the grace period + a PDB.

In depth:

t=0     pod marked for deletion
        ├─► SIGTERM to the app          (instant)
        └─► removal from EndpointSlice  (async: kube-proxy, LB — lag!)
t=0…3s  stale rules still send requests → connection refused / 5xx
fix:    preStop sleep 5 → traffic drains FIRST, then the process dies

1. preStop hook — sleep 5: gives the rules and LBs time to pull the pod out of rotation before SIGTERM.
2. SIGTERM in the app — stop accepting new connections, finish in-flight ones, exit.
3. terminationGracePeriodSeconds — longer than worst-case drain + in-flight time (default 30s, then SIGKILL).
4. PodDisruptionBudget — on node drains and upgrades (cordon + drain honors the PDB) it guarantees a minimum of live replicas.

⚠️ Common mistake: fixing only the app (a SIGTERM handler) and missing the endpoint race: without the preStop pause, requests keep arriving at a pod that has already begun shutting down.

Short answer: A Role (rules scoped to the namespace) + a RoleBinding to the team's group or users. For reuse — one ClusterRole with the standard permission set and a RoleBinding per namespace: the binding is namespaced, so the permissions are too. Pods authenticate with ServiceAccount tokens.

In depth:

1. Role vs ClusterRole — a Role lives in a namespace; a ClusterRole is global, but via a RoleBinding its permissions are narrowed to one namespace — the pattern for giving a dozen teams identical rights.
2. Subjects — bind to an IdP group (group: team-a) rather than to individuals by name.
3. ServiceAccount — a pod receives its SA's projected token; if the app needs API access, the SA gets bound with the same Role/RoleBinding.
4. Verify — don't guess, ask the cluster:

kubectl auth can-i list pods -n team-a --as u@corp --as-group team-a   # yes
kubectl auth can-i list pods -n team-b --as u@corp --as-group team-a   # no
kubectl auth can-i --list -n team-a --as u@corp    # the whole permission set at once

⚠️ Common mistake: handing out cluster-admin "temporarily, to skip the hassle." RBAC is additive — you can't subtract permissions later, only revoke bindings; start from least privilege.