11 DevOps Linux and Troubleshooting Interview Questions and Answers

Short answer: Not necessarily. On Linux, load average counts not only runnable processes but also tasks in uninterruptible sleep (D-state) — usually waiting on disk or NFS I/O. Load 30 with an idle CPU almost always means an I/O problem, not a shortage of cores.

In depth:

1. Linux quirk — unlike other Unixes, Linux load average includes D-state tasks. 30 processes stuck on a dead NFS share give you load 30 at zero CPU.
2. Find D-state — ps -eo state or top: the culprits show status D.
3. Check the disks — iostat -x: %util near 100 and high await = the disk is drowning.
4. Check NFS — a hung mount keeps processes in D-state indefinitely.

uptime                       # load: 30.2 29.8 28.5
top                          # %Cpu(s): ... 95 id → CPU is not the issue
ps -eo pid,state,wchan:30,comm | awk '$2=="D"'
iostat -x 1 3                # %util, await per disk
mount -t nfs,nfs4            # any NFS mounts, are they alive

⚠️ Common mistake: treating load average as a CPU metric and suggesting “add more cores”. If D-state tasks drive the load, a CPU upgrade changes nothing — the bottleneck is I/O.

Short answer: The kernel gives every process an oom_score — roughly the percentage of memory it uses — and kills the one with the highest. The score is shifted via oom_score_adj (−1000…+1000). The kill record lives in the kernel log: dmesg or journalctl -k.

In depth:

1. Where the trail is — the app gets SIGKILL and logs nothing; the kernel leaves the record: “Out of memory: Killed process 1234 (java)…”.
2. oom_score — roughly the share of memory used (RSS + swap): the more a process eats, the more attractive a victim it is.
3. oom_score_adj — a manual shift: −1000 removes the process from selection entirely, +1000 makes it the first candidate. Kubernetes sets −997 on Guaranteed pods.
4. cgroup limits — a memory limit on a cgroup localizes the OOM: the victim is picked inside the offending group, not among random neighbors on the host.

dmesg -T | grep -i 'killed process'
journalctl -k --since '1 hour ago' | grep -i oom
cat /proc/1234/oom_score /proc/1234/oom_score_adj
echo -900 > /proc/1234/oom_score_adj  # or OOMScoreAdjust=-900 in the systemd unit

⚠️ Common mistake: hunting for the crash cause in the application logs. SIGKILL cannot be caught — there will be silence; the kernel log is where to look.

Short answer: Most often you have either run out of inodes — space exists, but there is nowhere to write the new file’s metadata — or you have hit the root-reserved blocks: ext4 keeps ~5% unavailable to regular users by default.

In depth:

1. Inode exhaustion — df -i shows IUse% = 100%. The typical culprit is millions of tiny files: session caches, mail queues, build artifacts. The fix is finding and cleaning the breeding-ground directory; df -h never shows it by size.
2. Root reserve — df shows free gigabytes, yet a regular user gets ENOSPC while root writes fine — the telltale sign. Check with tune2fs -l, shrink the reserve with tune2fs -m 1.

df -h /data     # Use% 95% — space looks fine
df -i /data     # IUse% 100% → out of inodes
du --inodes -d1 /data | sort -n | tail   # where the millions of files live
sudo tune2fs -l /dev/sdb1 | grep -i 'reserved block'

⚠️ Common mistake: stopping at df -h and concluding “there is space, must be an app bug”. df -i is the mandatory second step for any ENOSPC.

Short answer: The file is still held open by a process: rm removes the name from the directory, but the data lives until the last file descriptor is closed. Find the culprit with lsof +L1; reclaim the space without a restart by truncating the file via /proc/<pid>/fd/N.

In depth:

1. Mechanics — space is freed when the link count is 0 AND nobody holds an fd. df counts such “deleted but open” files as used, while du no longer sees them — hence the classic df vs du discrepancy.
2. Diagnosis — lsof +L1 lists open files with link count 0: the process, the fd number, and the size.
3. Reclaim now — truncate through procfs: : > /proc/<pid>/fd/N zeroes the content without touching the process.
4. Prevent a repeat — copytruncate in logrotate, or signal the app to reopen its log (nginx: kill -USR1).

df -h /var/log && du -sh /var/log   # df sees 50 GB, du does not
lsof +L1 | grep deleted
# nginx 1234 ... 5w ... 53687091200 /var/log/access.log (deleted)
: > /proc/1234/fd/5                 # space is back, process keeps running

⚠️ Common mistake: restarting the service at peak hours for the sake of 50 GB. Truncating via /proc returns the space instantly with zero downtime.

Short answer: Almost certainly not. Linux deliberately fills “spare” memory with page cache — it is handed back to applications instantly. Look at the available column, not free; real memory pressure shows up as swap-in and major page faults.

In depth:

1. Empty memory is wasted memory — the kernel caches disk so repeated reads come from RAM instead of the device.
2. Real signs of pressure — growing si in vmstat 1 (swap-in), major page faults, OOM-killer entries in dmesg.

free -h       # watch available, not free
vmstat 1 5    # si/so columns — is active swapping happening

⚠️ Common mistake: “no free memory — let’s reboot” or dropping caches via drop_caches. Page cache is a feature: clearing it only slows the system down until it warms up again.

Short answer: A zombie is a child process that has exited but whose exit status the parent has not yet collected via wait(). All that remains is an entry in the process table: no code, no memory. That is why kill -9 is pointless — you are signaling something already dead.

In depth:

fork() → running → exit() → ZOMBIE (process-table entry)
                               │  parent calls wait()
                               ▼
                            entry removed (reaped)

1. The parent is always the cause — it fails to call wait()/waitpid() and ignores SIGCHLD. Fix or restart the parent: orphaned zombies get reparented to init/systemd, which reaps them immediately.
2. Zombies consume no resources — no CPU, no memory; just a PID and a table row.
3. When it becomes a problem — a mass zombie leak exhausts the PID space (kernel.pid_max): fork starts failing for every process on the host.

ps -eo pid,ppid,state,comm | awk '$3=="Z"'   # who is a zombie and who is the parent

⚠️ Common mistake: trying to “kill” a zombie with signals. The only way out is making the parent call wait() — or restarting the parent itself.

Short answer: D-state is uninterruptible sleep: the process is inside a system call, usually waiting on I/O (a dead NFS share, a failing disk). Signals — including SIGKILL — are delivered only when the syscall returns, so until the I/O completes or times out, the process cannot be killed.

In depth:

1. It is protection, not a bug — interrupting a process mid-operation with the kernel and hardware would leave data structures in an inconsistent state.
2. See where it is stuck — cat /proc/<pid>/stack (kernel stack) and the wchan column in ps: the kernel function the process sleeps in points at the subsystem — NFS, block layer.
3. Fix the cause, not the process — bring the NFS server back, umount -f / umount -l for a dead share, check the disk: dmesg for I/O errors, SMART.
4. If the I/O will never return — only a reboot releases the process.

ps -o pid,state,wchan:32,comm -p 1234   # which kernel function it sleeps in
cat /proc/1234/stack                    # kernel stack (requires root)
dmesg -T | tail -30                     # disk errors, NFS timeouts

⚠️ Common mistake: escalating kill → kill -9 → “why is it not working?!”. SIGKILL is powerless here by design — hunt for the stuck I/O, not for a stronger signal.

Short answer: strace is a syscall tracer: it answers “what is the process stuck on, which files and sockets does it touch”. perf is a sampling CPU profiler: it answers “where do the cycles go”. strace works via ptrace and slows the target dramatically — be careful in production; perf costs a few percent.

In depth:

strace -f -tt -T -p 1234    # live syscall stream with timings
strace -c -p 1234           # summary: counts and time per call
perf top -p 1234            # hot functions right now
perf record -g -p 1234 -- sleep 30 && perf report   # profile with stacks

⚠️ Common mistake: attaching strace to a loaded production service “just to look” — latency can grow by an order of magnitude. For “where does the CPU go”, perf first; strace — targeted and brief.

Short answer: First figure out which limit you hit: per-process (ulimit -n / LimitNOFILE) or the system-wide fs.file-max. Then count and classify the process’s descriptors — and find the leak. Raising the limit comes only after answering “why are there so many”.

In depth:

1. Which limit — each process has its own: cat /proc/<pid>/limits. For systemd services a shell ulimit does nothing — LimitNOFILE in the unit rules. The system ceiling is fs.file-max.
2. How many are open — ls /proc/<pid>/fd | wc -l against the process limit.
3. What exactly is open — lsof on the process: thousands of sockets in CLOSE_WAIT = the app is not closing connections; thousands of identical files = an fd leak in the code.
4. Only now the limit — if the growth is legitimate (traffic grew), raise LimitNOFILE and, if needed, fs.file-max.

cat /proc/1234/limits | grep 'open files'
ls /proc/1234/fd | wc -l
lsof -p 1234 | awk '{print $5}' | sort | uniq -c | sort -rn | head
systemctl show myapp -p LimitNOFILE
sysctl fs.file-max fs.file-nr

⚠️ Common mistake: silently raising the limit tenfold. If it is a descriptor leak, you have merely postponed the incident — and made it harder to investigate.

Short answer: iowait is time the CPU sits idle WHILE it has outstanding disk I/O. It is a form of idle, not work. High iowait says “the workload is waiting on the disk”, but by itself it does not prove the disk is the bottleneck: confirm via iostat -x.

In depth:

1. Definition — the kernel marks a tick as iowait when a CPU core is free but has a task blocked on I/O. If the CPU had other work, it would be doing it.
2. Corollary #1 — high iowait means CPU headroom: cycles are free, added compute load would run.
3. Corollary #2 — on a busy CPU a disk problem hides: iowait is low because the CPU is busy with other things, even though the disk is just as bad.
4. Confirming the disk — iostat -x: %util near 100, await well above the device’s baseline latency, a growing aqu-sz queue. Who generates the I/O — pidstat -d / iotop.

mpstat 1 5      # %iowait per core
iostat -x 1 5   # %util, r_await/w_await, aqu-sz
pidstat -d 1    # I/O per process

⚠️ Common mistake: reacting to the iowait number without iostat. iowait is a symptom of “workload waits on I/O while CPU is free”, not a disk-health metric.

Short answer: First reconcile df with du: if df sees usage that du does not, the space is held by deleted-but-open files or other mounts. Then narrow down the directory with du/ncdu, find files growing right now by mtime, and check the usual suspects: unrotated logs, journald, Docker layers.

In depth:

1. df vs du — lsof +L1 for deleted open files; make sure nothing is mounted over a directory, hiding old data underneath.
2. Narrow the directory — du -xh -d1 / | sort -h or ncdu -x: the -x flag keeps you from wandering onto other filesystems.
3. What is growing right now — find / -xdev -mmin -10 -size +100M: large files modified in the last 10 minutes are almost always the culprit.
4. Usual suspects — an app that suddenly enabled debug logging; journald without a cap (journalctl --disk-usage); container layers and logs (docker system df).

df -h / && du -xsh /                        # do the numbers agree
lsof +L1 | grep deleted
du -xh -d1 / 2>/dev/null | sort -h | tail
find / -xdev -mmin -10 -size +100M -ls 2>/dev/null
journalctl --disk-usage && docker system df

⚠️ Common mistake: deleting “something big” right away. Until the source of growth is found, freed space gets eaten back in minutes — find the writer first, clean up second.