10 Data Engineering Streaming Interview Questions and Answers

Short answer: Kafka is a distributed, append-only commit log. Producers append messages to the tail of a topic, consumers read them at their own pace, and the read position is tracked as an offset. A topic is split into partitions — the unit of parallelism and ordering.

In depth:

1. Topic — a named stream of events; logically a category of messages.
2. Partition — an ordered, immutable log within a topic. Each message gets a monotonically increasing offset (its position in the partition).
3. Producer writes to the tail of a partition; consumer reads sequentially and advances its own offset (commit), so reading does not delete data.
4. Broker — a cluster node that stores partitions; data lives per the retention policy, not "until first read".

Topic "orders"
 Partition 0: [0][1][2][3][4]───► new records appended at tail
 Partition 1: [0][1][2]
 Partition 2: [0][1][2][3]
                    ▲
              consumer offset = 2 (will read 3 next)

⚠️ Common mistake: assuming a consumer "takes" a message and it disappears like in a queue. In Kafka the log stays, many consumer groups read it independently, and only retention deletes data.

Short answer: The partition is the unit of parallelism: within a consumer group each partition is read by exactly one consumer. So effective parallelism is capped by the partition count — consumers beyond that sit idle.

In depth:

1. Consumer group — a set of consumers sharing a group.id, across which partitions are distributed (rebalance). Each partition is assigned to one consumer in the group.
2. Ordering is guaranteed only within a partition. More partitions → more parallelism, but no global ordering across the topic.
3. Relationship: consumers ≤ partitions for full utilization. With more consumers than partitions, the surplus stays idle.
4. Scaling: partitions set the ceiling. Their count is easy to increase but not decrease, and it changes how keys map to partitions.

⚠️ Common mistake: adding consumers to go faster while forgetting the ceiling is the partition count. Beyond it, consumers sit with no partitions assigned.

Short answer: Semantics are set by when you commit the offset and by producer configuration. Committing before processing gives at-most-once (can lose), after gives at-least-once (can duplicate), and an idempotent producer plus transactions give exactly-once.

In depth:

1. At-most-once — offset committed before processing. On a crash the message is already "read" but not processed → loss.
2. At-least-once — offset committed after successful processing. On failure the message is re-read → duplicates; the consumer must be idempotent.
3. Exactly-once (EOS) — enable.idempotence=true (broker-side dedup by producer id + sequence) plus transactions (transactional.id) that atomically tie writing the result to committing the offset.

⚠️ Common mistake: committing the offset before actually processing for "speed" — this silently loses data on any consumer crash.

Short answer: Ordering is guaranteed only within a single partition, not across the whole topic. The message key determines the partition: messages with the same key land in the same partition and are read in order.

In depth:

1. Ordering is per-partition. Within a partition offsets strictly increase; across partitions ordering is undefined.
2. Key-based routing: partition = hash(key) % num_partitions. One key → one partition → preserved relative order for that key.
3. No key (key = null) and the producer spreads messages across partitions (round-robin / sticky) — no ordering between them.
4. Practice: key by the entity whose order matters (e.g. user_id, order_id) so its events flow sequentially.

key="user-42" ─hash─► Partition 1  [event A][event B][event C]  (order preserved)
key="user-99" ─hash─► Partition 0
key=null      ─round-robin─► any partition

⚠️ Common mistake: expecting global ordering across the whole topic. With multiple partitions there is none — ordering holds only within a key/partition.

Short answer: Durability comes from replication: each partition is copied to several brokers (replication factor), and the ISR is the set of replicas caught up to the leader. The acks setting trades durability against latency, and retention/compaction decide how long data lives.

In depth:

1. Replication factor — how many copies of a partition are kept. RF=3 survives the loss of 2 brokers.
2. ISR (in-sync replicas) — replicas in sync with the leader. acks=all acknowledges a write only once all ISR have it.
3. Retention — data is kept by time (retention.ms) or size; consumers do not delete it.
4. Log compaction — an alternative: keep at least the latest value per key (a snapshot of state).

⚠️ Common mistake: setting acks=1 with a high RF and assuming data is safe. If the leader dies before replicating an acknowledged write, it is lost — for real guarantees use acks=all together with min.insync.replicas.

Short answer: A window groups events over time for aggregation. Tumbling windows are fixed, non-overlapping intervals; sliding windows overlap; session windows are defined by gaps in activity. Event-time uses when an event occurred; processing-time uses when it is processed.

In depth:

1. Tumbling — adjacent fixed-length windows with no overlap; each event in exactly one window (e.g. a count per minute).
2. Sliding — fixed-length windows advancing by a step smaller than the length; windows overlap and an event falls into several (moving average).
3. Session — a window closes after an inactivity gap; length depends on activity (user sessions).
4. Event-time vs processing-time: event-time yields correct results under delays and reordering but needs watermarks; processing-time is simpler and faster but skews results when there is lag.

Time axis ──────────────────────────►
Tumbling:  [ 0–1m ][ 1–2m ][ 2–3m ]
Sliding:   [ 0–1m ]
              [ 30s–1:30 ]
                 [ 1–2m ]
Session:   [a1 a2 a3]      (gap)      [b1 b2]

⚠️ Common mistake: aggregating by processing-time and being surprised by "wrong" daily totals when events arrive late. For accurate business metrics, compute on event-time.

Short answer: A watermark is an estimate that "we have probably received all events up to time T", letting the engine decide when to close an event-time window. Allowed lateness gives a window extra time to accept stragglers after the watermark has fired it.

In depth:

1. The problem: events arrive out of order and delayed (network, batching, mobile clients). On pure event-time it is unclear when a window's result is final.
2. Watermark advances through time and tells the engine "we no longer expect events with time < W" — windows close and emit on it.
3. Allowed lateness — the window keeps its state for some time past the watermark and recomputes the result if late events arrive.
4. Beyond the lateness bound an event goes to a side output (dead-letter) or is dropped — a trade-off between accuracy and latency/memory.

events by event-time:  e(10:00) e(10:02) e(09:59←late) e(10:03)
watermark ───────────────────────► 10:02
                       └ window 10:00–10:01 closed; e(09:59) within
                         allowed lateness → window recomputed

⚠️ Common mistake: setting an overly "strict" watermark for low latency and silently dropping late events — or the reverse, a huge allowed lateness that bloats in-memory state.

Short answer: CDC captures row-level changes from a database in real time. Debezium reads the transaction log (WAL/binlog) and publishes each insert/update/delete as an event to Kafka. Unlike periodic polling, CDC gives low latency, doesn't burden the DB with queries, and never misses intermediate changes.

In depth:

1. How it works: Debezium hooks into the database's replication log (Postgres WAL, MySQL binlog) and turns commits into a stream of events — no application changes.
2. Why better than polling: polling on updated_at misses intermediate states and deletes, loads the DB with heavy queries, and lags by the poll interval.
3. Guarantees: events flow in commit order per partition, typically with at-least-once semantics — the consumer must be idempotent (upsert by primary key).
4. Uses: syncing into a data lake/DWH, cache invalidation, event-driven microservice integration.

┌──────────┐  WAL/binlog  ┌──────────┐  events   ┌────────┐   ┌───────────┐
│ Postgres │ ───────────► │ Debezium │ ────────► │ Kafka  │ ─►│ Lake/DWH  │
└──────────┘              └──────────┘           └────────┘   └───────────┘

⚠️ Common mistake: replacing CDC with periodic SELECT ... WHERE updated_at > :last and losing deletes and intermediate row versions, while also loading the production database.

Short answer: Kafka fits a high-throughput, replayable event log with several independent consumers. SQS is a simple managed queue for distributing tasks without replay. Kinesis is a "Kafka-like" managed stream on AWS, and Pulsar is an alternative with multi-tenancy and separated compute/storage.

In depth:

• Kafka — high throughput, log retention and re-reads, many consumer groups, ecosystem (Connect, Streams). At the cost of operational complexity (or a managed offering like Confluent/MSK).
• SQS — a task queue: read-then-delete a message, no replay and no ordering (except FIFO), but near-zero operational cost.
• Kinesis / Pulsar — streaming with Kafka-like retention; Kinesis is AWS-native, Pulsar offers multi-tenancy and independent storage scaling.

⚠️ Common mistake: reaching for Kafka where a simple task queue with no replay is needed — you take on extra operational burden where SQS would have sufficed.

Short answer: Flink is a true streaming engine: it processes each event as it arrives, giving millisecond latency. Spark Structured Streaming has historically used micro-batches — it accumulates events over a short interval and processes them as a batch, which is simpler but adds delay.

In depth:

1. Processing model: Flink is event-at-a-time (true streaming); Spark uses micro-batches (mini-jobs every N ms/sec). Spark has a Continuous Processing mode, but the main path is micro-batch.
2. Latency: Flink is sub-second/milliseconds; Spark micro-batch is bounded by the batch interval.
3. State and time: Flink has rich state management, event-time, watermarks, and precise windows — its strong suit.
4. Ecosystem: Spark is convenient if the team already runs Spark for batch (one stack for batch + streaming); Flink is chosen when low latency and complex event logic are critical.

⚠️ Common mistake: calling Spark Structured Streaming "true" streaming on par with Flink. By default it is micro-batch, and latency is bounded by the batch interval.