11 DevOps Observability and SRE Interview Questions and Answers

Short answer: An SLI is a measured ratio: good events over total events. An SLO is an internal target for that SLI over a window (99.9% per 30 days). An SLA is an external customer contract where violations cost penalties. The error budget = 1 − SLO: the share of "allowed" errors you get to spend.

In depth:

SLO 99.9% over 30 days: 30 × 24 × 60 = 43,200 min
budget = 0.1% × 43,200 ≈ 43.2 min of downtime per month
SLO 99.99%              ≈ 4.4 min per month

The SLO is always stricter than the SLA: the internal target must trip before contractual penalties do.

⚠️ Common mistake: treating SLO and SLA as synonyms and not knowing the numbers. The interviewer expects the math: 99.9% monthly ≈ 43 minutes, 99.99% ≈ 4.4 minutes.

Short answer: The error budget is a decision mechanism, not a reporting number. The policy is agreed with product beforehand: budget exhausted → feature freeze, reliability work takes priority, releases are gated until the budget recovers. Without enforcement, all of it is theater.

In depth:

1. The policy is signed before the incident — the team and product agreed in advance: budget gone means features wait. Arguing at the moment of exhaustion is too late.
2. Escalation by remaining budget:

3. Reliability work gets priority — postmortem action items, rollback automation, tests: the things that actually win the budget back.
4. Recovery — the budget is computed over a rolling window (typically 30 days): old errors age out of the window and the budget regrows on its own.

⚠️ Common mistake: answering "we'll try harder." Without pre-agreed consequences, the error budget is just a dashboard number, not a contract between reliability and feature velocity.

Short answer: You page a human when users are hurting — that is, on SLIs: error rate, latency. A "CPU at 90%" alert wakes the on-call for non-problems and stays silent during real ones: a degradation has many possible causes but one symptom. Cause metrics belong on dashboards — for diagnosis.

In depth:

1. CPU 90% is not an incident — a batch job legitimately saturating a machine; the on-call was woken for nothing.
2. All boxes green, users hurting — a code bug, a bad config, a third-party API down: cause-based alerts stay silent.
3. A slow disk that hurts nobody — if the SLIs are fine, that is a ticket for tomorrow, not a 3 a.m. page.

⚠️ Common mistake: alerting on everything "just in case." The result is alert fatigue: the on-call mutes the channel and misses the real incident.

Short answer: Burn rate is how many times faster than budgeted you are spending the error budget: actual error rate ÷ budgeted error rate. Burn rate 1 means the budget is spent exactly by the end of the window. You alert on the burn speed over several windows: fast burns page, slow burns ticket. Every alert maps to the math of real user impact.

In depth:

SLO 99.9% over 30 days → budgeted error rate 0.1%
burn rate = actual error rate / 0.1%
burn 1 → budget spent exactly by the end of the 30-day window

page:   burn 14.4× over a 1 h window
        burned 14.4 × (1 h / 720 h) = 2% of the monthly budget in an hour
ticket: burn 6× over a 6 h window
        burned 6 × (6 / 720) = 5% of the budget — slow but steady

1. Multi-window, multi-burn-rate — a short window catches fast fires (page), a long one catches slow leaks (ticket); a short control window (5 min) clears the alert right after recovery.
2. Versus a static threshold — "error rate > 1%" either wakes you on a blip of a dozen requests or stays silent for weeks while a leak eats the budget. Burn rate scales urgency with severity by itself.
3. Answering "why was I woken?" — with numbers: "we burned 2% of the monthly budget in one hour."

⚠️ Common mistake: one threshold for every case. A fast fire and a slow leak need different windows and different urgency.

Short answer: Metrics are cheap aggregates: alerting and trends. Logs are discrete events: forensics for a specific incident, expensive at volume. Traces give per-request causality across services: "where did the 2 seconds go?" The signals are tied together via trace IDs and exemplars.

In depth:

1. Start with metrics — an SLI alert tells you something is wrong.
2. A trace shows where — which of the 12 services added 1.8 s of the 2.
3. Logs explain why — the stack trace, the exact request, the exact user.
4. Correlation — a shared trace ID in logs and exemplars in metrics let you jump between signals instead of eyeballing timestamps.

⚠️ Common mistake: "log everything — we'll sort it out later." In production that means a storage bill above your compute bill and grepping terabytes instead of an alert in seconds.

Short answer: Prometheus pulls: it scrapes /metrics endpoints on targets found via service discovery. A time series = metric name + label set. Counters only increase, so you look at speed, not value: rate() computes the per-second increase and correctly survives restarts.

In depth:

1. Pull, not push — Prometheus scrapes targets on a schedule: a successful scrape itself (up == 1) is a free health check; service discovery (Kubernetes, Consul) finds targets automatically.
2. Data model — http_requests_total{method="GET", status="500"}: every unique label combination is its own series.
3. rate() over counters — a counter's absolute value is meaningless (it depends on uptime); rate() gives increase/sec and handles counter resets on restart.

# 5xx ratio
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

# p99 from a histogram
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

4. rate() window ≥ 2× the scrape interval — otherwise the window holds fewer than two points and the graph has gaps.

⚠️ Common mistake: alerting on a raw counter value or taking rate() of a gauge. Counter → rate(), gauge → current value/delta.

Short answer: Cardinality is the number of unique label combinations: each combination is a separate time series living in memory. Memory grows roughly linearly with active series. A user_id label, a full URL, or a pod hash mints an unbounded number of series — and Prometheus OOMs.

In depth:

http_requests_total{path="/user/48211/cart", ...}
100,000 users × 5 methods × 10 statuses = 5,000,000 series
→ TSDB memory explodes, scrapes slow down, OOM

1. The rule — before adding a label, ask: could it have 10,000+ values? Then it is a dimension for traces and logs, not for metrics.
2. Usual suspects — user_id, session_id, full paths with parameters (normalize to /user/:id), IP addresses, pod hashes in label values.
3. Diagnosis — prometheus_tsdb_head_series and top metrics by cardinality via /api/v1/status/tsdb.
4. When many series are legitimate — the answer to scale is not "one more Prometheus" but VictoriaMetrics or Thanos: long-term storage, global querying, deduplication. A frequent question at Avito/Ozon interviews — that is their production stack.

⚠️ Common mistake: labeling by a request parameter "for easier filtering." One line of instrumentation — and a week later monitoring is down together with the service.

Short answer: Latency, traffic, errors, saturation — from Google's SRE book. This set is enough to monitor almost any user-facing system. The subtlety: measure latency on successful requests — errors usually respond fast and "improve" the average.

In depth:

1. Error latency tracked separately — a fast 500 mixed into the pool masks degradation: p99 "improves" while users suffer.
2. Saturation looks ahead — it is a leading signal: the queue grows now → latency degrades in minutes.
3. This is a blitz question — answer without pausing; follow-ups usually dig into latency and saturation.

⚠️ Common mistake: computing latency over all requests including errors — fast 5xx responses statistically hide the real slowdown.

Short answer: Mitigate first, diagnose second. The deploy correlation is obvious — roll back the release without hunting for root cause: while you debug, users are losing orders. Diagnosis comes after the bleeding is stopped.

In depth:

0–2 min    confirm: checkout SLI dashboard, blast radius (all? region? segment?)
2–5 min    correlate: the deploy 20 min ago is the prime suspect
5–8 min    ROLL BACK the deploy. Not a hotfix, not debugging — rollback
8–12 min   verify SLI recovery; communicate: incident channel,
           status page; if it is big — declare an incident
           and assign roles (IC / comms / ops)
12–15 min  preserve evidence: graphs, logs, release id — for the postmortem

1. Mitigate first — a rollback is almost always faster and cheaper than a diagnosis; the root cause can wait until morning.
2. Communication is not optional — a silent on-call creates a second incident; the channel timestamps later become the postmortem timeline.
3. Preserve state — capture logs and graphs before the rollback erases the picture.

⚠️ Common mistake: heroically debugging root cause while production burns. The interviewer is listening for the order: stop the damage → communicate → only then the cause.

Short answer: Blameless means focusing on the system, not on people: "human error" is where the analysis starts, not where it ends. What makes it useful: concrete action items with owners and deadlines that actually land in production, and findings shared across the whole organization.

In depth:

1. Why no blame — if incidents get people punished, people hide details, and the organization loses its learning data. The question is not "who pushed the button" but "why did the system let one push take down production."
2. Anatomy of a good postmortem:

3. Useful = action items closed — track them like regular sprint tasks; a postmortem after which nothing changed is a ritual.
4. Share widely — someone else's incident is the cheapest lesson for the neighboring team.

⚠️ Common mistake: "it was Dave's fault, he'll be more careful" — or 15 action items with no owners. Both guarantee a repeat incident.

Short answer: Toil is operational work that is all of: manual, repetitive, automatable, tactical, devoid of enduring value, and scaling linearly with service growth. Google caps toil at roughly 50% of SRE time — the rest must go into engineering that kills that toil.

In depth:

1. Six attributes — manual, repetitive, automatable, tactical (reactive), no enduring value, scales linearly with growth. The more that match, the purer the toil.
2. Measure first — one or two weeks of accounting: ticket categories, time per category. Without data, "we're drowning" never becomes a plan.
3. Attack the top — 2–3 categories usually make up half the volume: automation (a script instead of a runbook), self-service (a button for developers instead of a "grant me access" ticket), eliminating the source (fix the root bug).
4. The toil budget as an argument — "we spend 60% of our time on toil, here is the accounting" is a legitimate case to management for carving out engineering time.

⚠️ Common mistake: calling all ops work toil. On-call and investigating novel incidents are not toil: they require judgment and create knowledge.