Post cover image
Data Engineering Pipeline

July 11, 2026

The Most Dangerous Data Engineering Bugs Don’t Break the Pipeline

A red pipeline is easy to notice.

By Iqra Anwar

10 min read

An activity fails. A notebook throws an exception. A connection times out. The pipeline stops.

Someone investigates it.

But the most dangerous data engineering bugs are often not the ones that break the pipeline.

None

They are the ones that let it succeed.

The pipeline finishes with a green checkmark.

The data lands in the table.

The dashboard refreshes.

The report opens.

And the numbers are wrong.

That is a much bigger problem.

Because when a pipeline fails visibly, people know something needs attention.

When a pipeline succeeds technically but produces incorrect data, everyone assumes the result can be trusted.

And sometimes that false confidence lasts for days, weeks, or even months.

A Green Checkmark Only Means the Code Finished

One of the biggest lessons I learned while working with data pipelines is that technical success and data correctness are two completely different things.

A successful run usually means:

  • The connection worked.
  • The query executed.
  • The notebook completed.
  • The file was created.
  • The destination accepted the data.
  • No exception was raised.

That is useful.

But none of those checks answer questions like:

  • Did we load the correct month?
  • Did we duplicate existing records?
  • Did we exclude valid data?
  • Did a timestamp shift records into the wrong reporting period?
  • Did we count rows when we should have counted unique customers?
  • Did we apply the right business rule?
  • Did the source system change without telling us?

A pipeline can succeed perfectly while getting all of those things wrong.

And that is where data engineering becomes difficult.

Bug #1: The Pipeline Loaded the Wrong Data Correctly

Imagine a monthly pipeline that should load data from four months before the current month.

None

The business rule is:

June run   → Load February data
July run   → Load March data
August run → Load April data

The pipeline executes successfully.

The API responds.

The data is inserted.

The destination table contains new rows.

Everything looks fine.

Except the date logic accidentally subtracts three months instead of four.

So the July run loads April instead of March.

Technically, nothing failed.

The pipeline did exactly what the code told it to do.

The code was wrong.

This is an important distinction.

Computers are extremely good at executing incorrect instructions consistently.

That is why testing only whether a pipeline runs is not enough.

You also need to test whether it ran for the correct business period.

A simple validation might compare the expected month against the actual month found in the output:

expected_month = "2026-03"
actual_month = df.select("report_month").distinct().collect()
assert len(actual_month) == 1
assert actual_month[0]["report_month"] == expected_month

Without a validation like this, the pipeline may continue producing the wrong monthly data while remaining completely green.

Bug #2: The Row Count Matched, but the Report Was Still Wrong

None

Row count checks are useful.

They are also dangerous when treated as proof of correctness.

Suppose the source contains 50,000 rows.

The destination also contains 50,000 rows.

Looks good.

But what if:

  • 1,000 valid records are missing.
  • 1,000 duplicate records were inserted.

The total count still equals 50,000.

The pipeline passes the validation.

The data is still wrong.

This is why row counts should be treated as one validation signal, not the final answer.

A stronger validation process may include:

  • Total row count
  • Unique business key count
  • Duplicate count
  • Null count for critical columns
  • Min and max dates
  • Distribution by category
  • Total monetary value
  • Source-to-target reconciliation

For example:

SELECT
    COUNT(*) AS total_rows,
    COUNT(DISTINCT students) AS unique_students,
    MIN(ExamDate) AS earliest_date,
    MAX(ExamDate) AS latest_date
FROM ExamData;

Even this simple query tells us much more than COUNT(*) alone.

A pipeline that copies the expected number of rows is not necessarily a correct pipeline.

Bug #3: The Same Customer Appeared Multiple Times

None

This is one of those problems that looks simple until someone asks a basic business question:

How many customers completed a purchase?

Now imagine this data:

None

How many customers completed a purchase?

If we count rows where the status is Completed, the answer is 3.

If we count unique customers with at least one completed order, the answer is 2.

Both calculations are technically valid.

Only one matches the actual business requirement.

And this is exactly the kind of issue that does not break a pipeline.

The query runs.

The table loads.

The dashboard refreshes.

A number appears.

But the number may be answering the wrong question.

The technical part is easy:

df.filter(df["Status"] == "Completed").count()

The real question is whether the business actually wants:

df.filter(df["Status"] == "Completed") \
  .select("CustomerID") \
  .distinct() \
  .count()

That difference is not really about Python.

It is about grain.

Are we counting rows?

Orders?

Customers?

Transactions?

Subscriptions?

This is why one of the most important questions in data engineering is:

What exactly does one row represent?

Many serious data quality problems begin when the code is technically correct, but the grain of the metric is misunderstood.

Bug #4: A Timestamp Quietly Changed the Reporting Month

None

Date and timestamp issues are some of the easiest problems to miss.

Imagine a source system stores this timestamp:

2026-06-30 23:30:00

After timezone conversion, it becomes:

2026-07-01 04:30:00

Now a transaction that originally belonged to June appears in July.

The pipeline does not fail.

The timestamp is valid.

The transformation runs.

The report refreshes.

But the monthly totals no longer reconcile.

This becomes especially dangerous when reporting logic depends on expressions like:

MONTH(TransactionDate)

or:

WHERE TransactionDate >= '2026-06-01'
AND TransactionDate < '2026-07-01'

A timezone issue can quietly move records across days, months, quarters, or even years.

And the pipeline can still show a green checkmark.

This is why I no longer see timestamps as just a technical data type.

They can directly affect business reporting.

Whenever dates drive metrics, I want to know:

Which timezone is the source using?

Is the timestamp stored in UTC?

Which timezone does the business report in?

Should conversion happen before or after filtering?

Are we comparing timestamps or calendar dates?

Could daylight-saving changes affect the result?

A single incorrect timezone assumption can make an otherwise perfectly running pipeline produce the wrong business numbers.

Bug #5: The Pipeline Reran and Duplicated Everything

None

Imagine a pipeline loads 20,000 records.

The data is successfully written to the destination.

Then another activity fails before the overall pipeline finishes.

The run is marked as unsuccessful.

Someone reruns it.

The same 20,000 records are inserted again.

Now the destination contains 40,000 rows.

The rerun succeeded.

The problem is that the pipeline was not designed to be safely rerun.

This is where idempotency becomes important.

An idempotent pipeline should produce the same correct result even when the same run is executed more than once.

There are several ways to achieve this:

Use a unique business key.

Use a MERGE instead of blindly appending data.

Delete and reload only the required partition.

Track already processed files.

Maintain checkpoints.

For example:

MERGE INTO TargetTable AS target
USING SourceTable AS source
ON target.RecordID = source.RecordID
WHEN MATCHED THEN
    UPDATE SET
        target.Status = source.Status,
        target.ModifiedDate = source.ModifiedDate
WHEN NOT MATCHED THEN
    INSERT (RecordID, Status, ModifiedDate)
    VALUES (source.RecordID, source.Status, source.ModifiedDate);

The goal is simple:

A rerun should not corrupt the data.

A pipeline that works correctly only when everything succeeds on the first attempt is fragile.

Production pipelines need to expect retries, failures, partial loads, and unexpected interruptions.

Because sooner or later, they will happen.

Bug #6: A New Source Value Broke the Business Logic Without Breaking the Code

None

Consider a status column with these expected values:

Completed
Cancelled
Pending
Processing
Refunded

The transformation contains:

df = df.withColumn(
    "Is_Completed",
    when(col("Status") == "Completed", 1).otherwise(0)
)

Later, the source system introduces:

Complete

or:

COMPLETED

or even:

Completed

with a trailing space.

The pipeline does not fail.

The new value simply gets classified as zero.

Now completed transactions are silently undercounted.

This is what makes these bugs dangerous.

Nothing crashes.

Nothing turns red.

The data is simply wrong.

That is why unknown categories should not always be allowed to pass quietly.

Instead of only writing transformation logic, it is also important to inspect the values actually arriving from the source:

df.select("Status").distinct().show()

Or better, validate them against an approved list:

valid_statuses = [
    "Completed",
    "Cancelled",
    "Pending",
    "Processing",
    "Refunded"
]
invalid_df = df.filter(~col("Status").isin(valid_statuses))

If unexpected values appear, they should be reviewed.

Otherwise, a small source-system change can quietly alter KPIs while every pipeline activity continues to succeed.

Bug #7: The Transformation Was Correct, but the Requirement Was Wrong

None

This is probably the hardest type of bug.

You receive a requirement:

Count all completed orders.

You implement it.

The code is tested.

The pipeline runs successfully.

Later, someone says:

Actually, we need the number of unique customers who completed at least one eligible order during the quarter, excluding test accounts and cancelled returns.

That is not really a code bug.

It is a requirement-definition problem.

But the result is the same:

The reported number is wrong.

This happens because business metrics often sound simple until you start asking detailed questions.

Take the word completed.

Does it mean:

Status equals Completed?

At least one completed order per customer?

Only the latest order status?

Completed within the reporting period?

Completed and paid?

Completed excluding internal accounts?

Completed for selected product categories only?

One word can hide an entire page of business logic.

That is why some of the most important questions in data engineering are not technical.

They are questions like:

What exactly are we counting?

At what grain?

Over what time period?

What should be excluded?

What happens when the same entity appears multiple times?

Which source is considered the source of truth?

Getting those answers early can prevent far more bugs than writing better code later.

The Dangerous Part Is That the Dashboard Looks Normal

Bad data does not always look obviously bad.

Imagine yesterday's revenue was 1.2 million and today's dashboard shows 1.19 million.

Nobody panics.

Maybe that is correct.

Or maybe hundreds of transactions are missing.

Suppose a conversion rate changes from 72% to 69%.

That may still look believable.

But perhaps duplicate records suddenly increased the denominator.

The hardest data bugs often produce numbers that look completely reasonable.

And that is exactly why they survive.

A dashboard showing zero revenue will immediately raise questions.

A dashboard showing revenue that is wrong by 4% may not.

This is why visual inspection alone is weak validation.

A number can look perfectly normal and still be completely wrong.

Good validation should compare the output against something meaningful:

Previous periods

Source-system totals

Known sample records

Expected category distributions

Business-controlled reports

Independent calculations

The real question should not be:

Does this number look okay?

It should be:

What evidence do we have that this number is correct?

What I Check Now Before Trusting a Pipeline

I no longer consider a pipeline validated just because it runs successfully.

I want to know:

Did the expected records arrive?

I compare the source and destination.

Did anything duplicate?

I check business keys and duplicate counts.

Are critical fields suddenly null?

I monitor null values in important columns.

Are the dates correct?

I inspect minimum dates, maximum dates, and reporting periods.

Did any new category appear?

I review distinct values in important status and type columns.

Can the pipeline be safely rerun?

I test it.

Do important totals reconcile?

I compare business metrics across the source, transformed data, and final reporting layer.

Can I manually trace a few records from source to final report?

This is one of my favorite checks.

Pick a few real records and follow them through the entire path:

Source System
      ↓
Raw Data
      ↓
Transformation
      ↓
Curated Table
      ↓
Semantic Model
      ↓
Dashboard

That simple exercise can expose problems that aggregate checks miss.

Sometimes the total row count looks correct while individual records are being transformed incorrectly.

Tracing a small sample end to end can reveal that quickly.

Monitoring Pipeline Health Is Not the Same as Monitoring Data Health

This distinction matters.

Pipeline monitoring may tell you:

Status: Success
Duration: 4 minutes 32 seconds
Rows copied: 85,402
Errors: 0

Useful.

But data-health monitoring should tell you things like:

Duplicate Order IDs: 312
Unexpected Status Values: 2
Null Customer IDs: 47
Source Count Difference: -1,205
Latest Available Date: 2026-06-29
Expected Latest Date: 2026-06-30

Those are two completely different types of observability.

One tells you whether the pipeline executed.

The other tells you whether the data can be trusted.

A reliable data platform needs both.

Because a pipeline can be healthy while the data is unhealthy.

The Best Data Engineers Are Professionally Suspicious

Not cynical.

Not negative.

Just careful.

They do not assume that because the query executed, the answer is correct.

They ask:

Why did this count increase?

Why did this month suddenly drop?

Why is the source missing records?

Why does one customer have six rows?

Why are these timestamps one day ahead?

Why did the same file process twice?

Why does the dashboard disagree with the source?

Why did a new status appear this week?

Sometimes the answer is harmless.

Sometimes that one question catches a serious issue before it reaches leadership, customers, finance teams, or external reports.

I have started to believe that a data engineer should never blindly trust a green checkmark.

Trust should come from validation.

Final Thoughts

The most obvious data engineering failures are usually the easiest ones to detect.

A connection fails.

A notebook crashes.

A file is missing.

An exception is thrown.

The pipeline turns red.

The dangerous failures are quieter.

The wrong month loads successfully.

Duplicates are inserted without errors.

A new status is silently ignored.

A timestamp moves into the next reporting period.

A KPI counts rows instead of customers.

The dashboard refreshes.

Everything turns green.

And nobody realizes the data is wrong.

That is why successful execution should never be confused with correct output.

A good pipeline should not only run.

It should produce the right data, at the right grain, for the right period, under the right business rules — and give us enough evidence to trust the result.

Because in data engineering, the scariest sentence is not:

"The pipeline failed."

It is:

"The pipeline succeeded, but the data was wrong."