Data Engineering Interview Prep Series — Part 6: Data Quality & Observability Questions You Must…

Why Data Quality Matters in Interviews

Businesses lose millions due to:

Broken data pipelines
Wrong dashboards
Incorrect ML predictions
Delays in executive decisions

A pipeline that runs fast but produces wrong data is worse than a pipeline that fails.

These questions help interviewers assess whether you build trustworthy data.

Q1 — What Are the Dimensions of Data Quality?

Expected answer includes:

Accuracy
Completeness
Freshness
Consistency
Validity
Uniqueness
Lineage visibility

Bonus: mention business SLAs and "fitness for purpose."

Q2 — How Do You Enforce Data Quality in ETL?

Must include:

✔ Null checks ✔ Referential integrity checks ✔ Deduplication ✔ Schema validation ✔ Data drift alerts ✔ Record count reconciliation

Interviewer loves the phrase:

"Data quality gates before writing to production zones."

Q3 — Code Example: Basic Quality Validation in PySpark

from pyspark.sql import functions as F
df_valid = df.filter(F.col("customer_id").isNotNull()) \
             .filter(F.col("amount") > 0)
df_bad = df.exceptAll(df_valid)

from pyspark.sql import functions as F
df_valid = df.filter(F.col("customer_id").isNotNull()) \
             .filter(F.col("amount") > 0)
df_bad = df.exceptAll(df_valid)

Show how bad data gets isolated → not lost.

Q4 — How Do You Handle Schema Drift?

Expected methods:

✔ Schema registry (Iceberg, Delta, Confluent) ✔ Automated detection + notification ✔ Backward-compatible evolution ✔ Mapping docs & versioning

Mention governance + risk control.

Q5 — What Is Observability in Data Pipelines?

Correct definition:

Continuous monitoring of data health, not just job failures.

Highlight key observability pillars:

Data quality alerts
Pipeline lineage
Change tracking
SLA monitoring
Logging + metrics + anomaly detection

Q6 — How Do You Set Data SLAs?

Real examples:

Data freshness: updated every 15 minutes
Availability: 99.9% uptime
Pipeline latency: < 3 mins from event → dashboard

Include alerting when SLA breached.

Q7 — Detecting Upstream Data Failures

Interviewer wants proactive mindset:

✔ Unexpected null spike alert ✔ Count mismatch detection ✔ Schema mismatch trigger ✔ Distribution shift detection (AI-based) ✔ Quality dashboards for stakeholders

Q8 — CDC Pipelines: Preventing Logical Duplicates

Solution patterns:

Idempotent loads
Merge with natural unique keys
Change type validation (I/U/D)
Sequence timestamps

"Duplicates are a symptom. Idempotency is the cure."

Q9 — Data Lineage: Why It Exists

Explain:

✔ Impact analysis (who will break if upstream fails?) ✔ Debugging pipeline issues ✔ Governance for regulated industries (e.g. healthcare)

Bonus: mention OpenLineage, DataHub, Atlan, Amundsen.

Q10 — Python Check for Unexpected Value Distribution

Quick interview-style snippet:

from great_expectations.dataset import PandasDataset
dataset = PandasDataset(df)
dataset.expect_column_values_to_be_between("age", 0, 120)

from great_expectations.dataset import PandasDataset
dataset = PandasDataset(df)
dataset.expect_column_values_to_be_between("age", 0, 120)

Shows tool knowledge + automation mindset.

What Interviewers Look For

✔ You prevent bad data — not just react to it ✔ You speak both business & technical language ✔ You design automated guardrails ✔ You care about trust, not just transformation

The best data engineers protect business credibility.

Part 7 Coming Next…

Business & Behavioral Questions — What Data Engineers Must Answer Beyond Tech Includes: ✨ Stakeholder communication scenarios ✨ Incident management questions ✨ Reporting business impact ✨ Conflict handling in data teams ✨ Productivity & ownership traits

Contents