8 Data Engineering Architecture Interview Questions and Answers

Short answer: First I clarify requirements (volume, arrival rate, required latency, SLAs, access patterns), and only then walk the layers: ingestion → storage → processing → serving, adding quality and cost. Tools come last.

In depth:

1. Clarify requirements — volume (GB/TB/PB per day), velocity (events/sec), read latency needed (seconds vs hours), who the consumer is (BI analyst, ML, application), consistency and retention needs.
2. Ingestion — batch or streaming, pull or push, a buffer (Kafka) to decouple producers from consumers, schema and its evolution.
3. Storage — a raw layer in the data lake (cheap, any data), marts in a warehouse/lakehouse; format choice (Parquet), partitioning.
4. Processing — transforms (dbt/Spark/Flink), the data model (star, medallion), idempotency and late-data handling.
5. Serving — BI over the warehouse, a low-latency OLAP/KV store for apps, reverse ETL into operational systems.
6. Quality and cost — data tests, contracts, freshness monitoring; sizing the cost of scanned bytes and idle compute.

Requirements (volume · latency · SLA · access)
        │
        ▼
 Ingest ──► Storage ──► Processing ──► Serving
(Kafka)   (lake →       (dbt/Spark/   (BI · OLAP ·
          warehouse)     Flink)        reverse-ETL)
        └──── data quality · cost ────┘

⚠️ Common mistake: jumping straight to tools ("let's use Kafka and Spark") before establishing volume, latency and access patterns — a daily batch doesn't need streaming.

Short answer: Lambda keeps two paths — a batch layer for accurate full recomputes and a speed layer for freshness, merged at serving; Kappa drops batch and treats everything as one stream, replaying the log when needed. Kappa is simpler to operate, Lambda gives a robust batch recompute.

In depth:

• Lambda — three layers: batch layer (full recompute over the whole dataset, source of truth), speed layer (stream for recent data), serving layer (merges both). Downside — logic duplicated across two codebases.
• Kappa — streaming only: everything is written to a durable log (Kafka), processing is single, history comes from replaying the log. Downside — heavy historical recomputes and large windows strain the stream engine.
• When Lambda — you have heavy batch analytics where accuracy and full reprocessing matter, and the stream only adds freshness.
• When Kappa — a streaming-first product, one logic path, an engine (Flink) that handles replay too; lower operational overhead.

Lambda:  source ─┬─► batch layer ─┐
                 └─► speed layer ─┴─► serving

Kappa:   source ──► log (Kafka) ──► stream processing ──► serving
                        ▲ replay │

⚠️ Common mistake: dragging in Lambda with two codebases where Kappa suffices — the duplicated logic drifts and produces different numbers in batch and stream.

Short answer: An SDK/collector sends events into Kafka as a buffer that decouples producers from consumers; fix the schema (Avro/Protobuf) in a Schema Registry, partition by key, and from the topic land into the raw lake layer and onward to the warehouse. Kafka absorbs spikes and enables replay.

In depth:

1. Collection — a client SDK/edge collector batches events, sends over HTTP/gRPC; validate and enrich (geo, user-agent) on ingress.
2. Buffer — Kafka — decouples producer and consumer rates, absorbs bursts, enables replay. Partition by user_id/session_id for per-key ordering.
3. Schema — Avro/Protobuf + Schema Registry, backward-compatible evolution; malformed events go to a dead-letter topic.
4. Landing in the lake — a consumer (Kafka Connect/Flink) writes to object storage as Parquet, partitioned by date/hour; this is the raw (bronze) layer.
5. Onward — process into a curated layer and load marts into the warehouse; idempotency by event_id, deduplication.

SDK/collector ─► Kafka (partitions by user_id) ─► consumer
                   │  replay, buffer                 │
                   │                                 ▼
             Schema Registry           lake (Parquet, /dt=YYYY-MM-DD/hh)
                                                    │
                                                    ▼
                                          warehouse / marts

⚠️ Common mistake: writing events straight into the warehouse synchronously from the app — the backend falls over under a traffic spike; a buffer (Kafka) is mandatory to decouple and smooth peaks.

Short answer: The choice is driven by required latency and the value of freshness: if the business is fine with data once an hour/day — batch (cheaper, simpler); if decisions are made in seconds (fraud, alerts, personalization) — streaming. A modern platform combines both: Kafka + warehouse + dbt for batch and Flink for streaming.

In depth:

• Batch — scheduled processing (ELT: Fivetran/Airbyte → warehouse → dbt). Cheap, easy to test and recompute, simpler idempotency. Latency — minutes to hours.
• Streaming — event-by-event / micro-window processing (Flink, Kafka Streams). Latency — sub-second; harder: state, late data, exactly-once.
• Decision rule — start from "what is the cost of data latency?". No value in sub-second freshness → batch.
• Hybrid — a single durable log (Kafka) feeds both the stream path (Flink for realtime marts) and the batch path (dump to lake/warehouse, dbt models).

⚠️ Common mistake: building streaming "for the future" when a daily batch is enough — you pay for complexity and state without gaining any value from freshness.

Short answer: A lakehouse puts data in object storage (S3/GCS) in an open table format (Iceberg/Delta/Hudi) that adds ACID transactions, schema evolution and time travel on top of files, with a query engine (Trino/Spark/Databricks) on top. Organize the layers by medallion: bronze → silver → gold.

In depth:

1. Storage — object storage: cheap, effectively infinitely scalable, separates storage from compute.
2. Table format — Iceberg/Delta/Hudi over Parquet: ACID, snapshot isolation, schema/partition evolution, time travel, small-file compaction.
3. Layers (medallion) — bronze (raw data as-is), silver (cleaned, typed, deduped), gold (aggregates and business marts).
4. Engine — Trino/Spark/Databricks read the same format; BI hits gold. One copy of data for SQL, ML and streaming.
5. Governance — a catalog (Glue/Unity), access control, evolution without rewriting data.

        Object storage (S3/GCS)
   ┌───────────┬───────────┬───────────┐
   │  bronze   │  silver   │   gold    │ ← Iceberg/Delta (ACID, time travel)
   │   raw     │  cleaned  │   marts   │
   └───────────┴───────────┴───────────┘
        ▲ ingest        ▲ Trino/Spark/dbt ─► BI · ML

⚠️ Common mistake: dumping everything into a "data lake" of bare Parquet files with no table format — you get a data swamp with no ACID, no schema evolution and a small-files problem.

Short answer: Separate storage and compute so you pay for them independently; the biggest cost drivers are scanned bytes and idle compute. You manage these with partitioning, clustering, columnar formats and auto-suspend of clusters.

In depth:

1. Separate storage and compute — storage is cheap and grows independently; compute spins up under load and shuts down. Scaling is horizontal.
2. Fewer scanned bytes — partitioning by date/key, partition pruning, clustering, columnar Parquet, reading only needed columns. In pay-per-scan (BigQuery) this is direct money.
3. Idle compute — auto-suspend warehouses/clusters, right-sizing, spot instances for batch, separate warehouses per workload.
4. Materialization — pre-aggregates/incremental models instead of recomputing everything; result caching.
5. FinOps — cost monitoring per query/team, quotas, cost allocation.

⚠️ Common mistake: ignoring scanned-byte volume — SELECT * over an unpartitioned table on a pay-per-scan engine burns the budget and doesn't scale.

Short answer: Match the serving store to the access pattern: a warehouse (Snowflake/BigQuery) for BI and analytical scans; a low-latency OLAP store (ClickHouse/Druid) for interactive dashboards and in-app analytics; key-value for point lookups; and reverse ETL pushes data back into operational systems (CRM, etc.).

In depth:

• Warehouse — columnar, for complex analytical queries and BI; latency in seconds, limited concurrency. Not for user-facing queries at thousands of RPS.
• Low-latency OLAP — ClickHouse/Druid/Pinot: sub-second aggregates at high concurrency, for in-app analytics and realtime dashboards.
• Key-value / cache — Redis/DynamoDB for point reads by key (profile, ML features) at millisecond latency.
• Reverse ETL — from the warehouse back into SaaS/operational systems (Hightouch/Census) so sales and marketing can use the data.

⚠️ Common mistake: putting a user-facing feature with thousands of requests per second directly on an analytical warehouse — it isn't built for high concurrency and low latency; you need an OLAP or KV layer.

Short answer: The modern data stack is a set of managed, modular services: connectors (Fivetran/Airbyte) → cloud warehouse/lakehouse → transforms (dbt) → BI, plus reverse ETL and orchestration. Buying off the shelf is worth it for speed; building is for when volume/specifics make a vendor too expensive or impossible.

In depth:

1. Stack layers — extract-load (Fivetran/Airbyte), storage (Snowflake/BigQuery/lakehouse), transform (dbt), BI (Looker/Metabase), orchestration (Airflow/Dagster), reverse ETL.
2. Buy — fast time-to-value, ready-made connectors, less team needed for upkeep; downside — cost grows with volume, less control.
3. Build — for large volumes/nonstandard sources and strict requirements; downside — you need a team, it takes longer, you own the operations.
4. In practice — buy at the start (managed ELT + warehouse + dbt), custom-build only the bottlenecks (e.g. your own ingest for extreme clickstream), don't build everything from scratch.

⚠️ Common mistake: building your own connectors and orchestration "to save money" while volumes are small — early on a managed stack is almost always cheaper in total cost of ownership than staffing a team to maintain homegrown tooling.