Cache Solved Our Database Problem… Then Created a New One

Sometimes it is.

But many production issues don't happen before cache. They start after cache.

We once reduced database load dramatically.

The application became faster.

Infrastructure cost dropped.

Everything looked perfect…

Until users started seeing old data.

That's when we realized:

Cache doesn't remove complexity. It simply moves it somewhere else.

Before Cache: What Problems Were We Facing?

Every request was going directly to the database.

As traffic increased, so did the pressure on the DB.

Typical problems

• High database CPU usage
• Slow API response time
• More connection timeouts
• Expensive database scaling
• Peak traffic affected everyone

Example:

1000 users
↓
1000 database queries

1000 users
↓
1000 database queries

The database became the bottleneck.

After Adding Cache: What Improved?

We introduced a cache layer for frequently accessed data.

Instead of asking the database every time:

Application
↓
Cache
↓
Database (only when needed)

Application
↓
Cache
↓
Database (only when needed)

The result was immediate.

Benefits

✅ Faster response time

✅ Lower database load

✅ Better user experience

✅ Lower infrastructure cost

✅ More stable during traffic spikes

Example:

Instead of:

1000 DB queries

1000 DB queries

We now had:

950 Cache hits

50 DB queries

950 Cache hits

50 DB queries

The improvement was huge.

Everyone celebrated.

But only for a while.

The New Problems Started

Performance improved.

Data accuracy became the new challenge.

Users began reporting strange issues.

Examples:

• Updated profile still showing old name
• Product price not changing immediately
• Deleted item still visible
• Dashboard showing old numbers
• Different users seeing different values

Nothing was actually broken.

The cache was simply serving old data.

Why Does This Happen?

A cache stores a copy of data.

When the database changes…

The cache may still contain the old version.

Example:

Database
Price = $120

Database
Price = $120

Cache

Price = $100

Price = $100

If the application reads from cache first…

The customer still sees $100.

This is called stale data.

Sometimes stale data is acceptable.

Sometimes it causes serious business problems.

New Edge Cases We Had to Handle

Adding cache introduced many new decisions.

1. Cache Invalidation

When should old cache be removed?

• After update?
• After delete?
• After insert?

One missed invalidation can create incorrect data.

2. Cache Expiry (TTL)

If TTL is too long

• Users see outdated data

If TTL is too short

• Database load increases again

Finding the right balance is important.

3. Cache Miss

What happens if data is not in cache?

Typical flow:

• Read cache
• Cache miss
• Read database
• Save into cache
• Return response

Simple on paper.

Important in production.

4. Cache Stampede

Imagine the cache expires.

Suddenly:

1000 requests arrive at the same time.

Every request goes to the database.

The database gets overloaded again.

Possible solutions:

• Request locking
• Background refresh
• Random TTL
• Single-flight requests

5. Updating Multiple Services

Modern systems often have multiple services.

One service updates the database.

Another service still has old cache.

Now different services return different data.

Keeping everything synchronized becomes harder.

6. What Should Never Be Cached?

Not everything belongs in cache.

Be careful with:

• Payment status
• Bank balance
• OTP
• Authentication tokens
• Real-time stock availability

Showing old data here can create real business issues.

A Simple Real Example

Imagine an e-commerce application.

A customer updates their delivery address.

Database:

New Address

New Address

Cache:

Old Address

Old Address

The next order still uses the old address.

The API worked.

The database was correct.

The cache caused the mistake.

These bugs are difficult because everything looks healthy from the outside.

Lessons We Learned

Cache is not just about speed.

It is also about consistency.

Before adding cache, ask yourself:

• What data can become stale?
• When should cache expire?
• How will cache be refreshed?
• What happens after an update?
• What happens during heavy traffic?
• Is eventual consistency acceptable?

These questions are often more important than choosing the cache technology.

Final Thoughts

Cache is one of the best tools for improving performance.

But every performance optimization introduces new trade-offs.

Before cache, we worried about database load.

After cache, we worried about data consistency.

The database stopped being the bottleneck.

Keeping cache and database in sync became the real engineering challenge.

Good engineers know how to add cache.

Great engineers also know how to keep cached data correct.