The Day Stale Cache Data Cost Us Three Hours of Debugging

Phil Karlton's famous quote, "there are only two hard things in Computer Science: cache invalidation and naming things", gets repeated so often it's become a joke. Which is unfortunate, because the first one is genuinely, actually hard in ways that take real incidents to appreciate.

I had my incident.

What We Were Caching

User profile data. Public-facing profile pages: display name, bio, avatar URL, follower count. The data was fetched on every page load, consistent enough that it was a natural caching candidate, and our database was showing the cost of repeated reads.

Simple cache-aside pattern:

async function getUserProfile(userId: string) {
  const cacheKey = `profile:${userId}`;

  // Check cache first
  const cached = await redis.get(cacheKey);
  if (cached) return JSON.parse(cached);

  // Cache miss, fetch from DB
  const profile = await db.user.findUnique({
    where: { id: userId },
    select: { id: true, name: true, bio: true, avatarUrl: true, followersCount: true },
  });

  // Store in cache with 1 hour TTL
  await redis.setex(cacheKey, 3600, JSON.stringify(profile));

  return profile;
}

When the user updated their profile, we invalidated:

async function updateUserProfile(userId: string, data: UpdateProfileInput) {
  const updated = await db.user.update({ where: { id: userId }, data });
  await redis.del(`profile:${userId}`); // invalidate cache
  return updated;
}

Clean. Obvious. Deployed. Worked fine in testing.

The Incident

A user updated their display name. They refreshed the page. Old name. Refreshed again. Old name. Filed a support ticket.

We checked the database, the new name was there. We checked Redis, the cache had the old name. The TTL was still showing 3540 seconds. The DEL had run (we could see it in logs) and yet the old value was back in cache.

Here's what happened.

The Race Condition

We had scaled to multiple server instances behind a load balancer. Two instances were involved:

Timeline:
─────────────────────────────────────────────────────────────
Server A (handling profile update):
  1. db.user.update()         ← writes new name to database
  2. redis.del('profile:42')  ← deletes cache key

Server B (handling profile page request, happened in the 2ms gap):
  1. redis.get('profile:42')  ← CACHE MISS (key was just deleted by A)
  2. db.user.findUnique()     ← BUT READS OLD DATA (replication lag, read replica was behind)
  3. redis.setex('profile:42', 3600, OLD_DATA)  ← writes OLD data back to cache
─────────────────────────────────────────────────────────────

The write happened on the primary. We were reading from a read replica. The replica was ~200ms behind at the time of the read. Server B hit the cache miss right as Server A deleted the key, read stale data from the replica, and cached that stale data for another hour.

The delete-then-populate race is a known problem with cache-aside. You can't avoid it completely with basic invalidation, the window is small but under load it's not rare.

Why "Update Cache Instead of Delete" Makes It Worse

The naive fix people reach for: instead of deleting the cache key, update it with the new value.

async function updateUserProfile(userId: string, data: UpdateProfileInput) {
  const updated = await db.user.update({ where: { id: userId }, data });

  // Update cache directly instead of deleting
  await redis.setex(`profile:${userId}`, 3600, JSON.stringify(updated));

  return updated;
}

This creates a worse race condition:

Server A: db.user.update()          ← writes v2
Server B: db.user.update()          ← writes v3 (user changed their mind)
Server B: redis.setex(v3 data)      ← sets cache to v3
Server A: redis.setex(v2 data)      ← overwrites cache with v2 ← WRONG

Because HTTP requests don't complete in order, the later write's cache update can arrive before the earlier one. Database has v3 (correct). Cache has v2 (stale). This state persists until TTL expires.

Update-cache is strictly worse than delete-cache for concurrent writes.

Fix 1: Read From Primary After Miss

Simplest fix for the replica lag problem: after a cache miss, always read from the primary.

async function getUserProfile(userId: string) {
  const cached = await redis.get(`profile:${userId}`);
  if (cached) return JSON.parse(cached);

  // Cache miss: read from primary, not replica
  const profile = await db.$primary().user.findUnique({
    where: { id: userId },
    select: { id: true, name: true, bio: true, avatarUrl: true },
  });

  await redis.setex(`profile:${userId}`, 3600, JSON.stringify(profile));
  return profile;
}

This adds load to your primary on cache misses, but cache misses should be infrequent. For a profile with a 1-hour TTL, a miss happens once per hour per user, fine.

If you don't have a way to route reads to primary specifically, ensure cache misses go through a connection that bypasses the read replica pool.

Fix 2: Short TTL as a Safety Net

Any cache that relies only on active invalidation is one missed invalidation away from serving stale data indefinitely. Networks fail. Code has bugs. Deploys happen.

A TTL is your recovery mechanism:

// Don't set a 1-hour TTL for user-facing data that updates frequently
// 5 minutes means stale data self-heals at most every 5 minutes
await redis.setex(`profile:${userId}`, 300, JSON.stringify(profile));

Yes, more cache misses. Yes, more database reads. That's the tradeoff. For most user-facing data, "eventually consistent within 5 minutes" is acceptable. For financial data or inventory, even 5 seconds may be too long, at which point you need to reconsider whether caching is right for that data.

Set your TTL to the longest staleness your users would not notice or care about.

Fix 3: Cache Versioning

For data that must be consistent immediately after writes, use a version or timestamp in the cache key:

async function getUserProfile(userId: string) {
  // Get the user's current version from a fast, always-fresh source
  const version = await redis.get(`profile:${userId}:version`);

  if (version) {
    const cached = await redis.get(`profile:${userId}:v${version}`);
    if (cached) return JSON.parse(cached);
  }

  const profile = await db.user.findUnique({ where: { id: userId }, ... });

  const newVersion = Date.now();
  await redis.setex(`profile:${userId}:version`, 3600, newVersion.toString());
  await redis.setex(`profile:${userId}:v${newVersion}`, 3600, JSON.stringify(profile));

  return profile;
}

async function updateUserProfile(userId: string, data: UpdateProfileInput) {
  const updated = await db.user.update({ where: { id: userId }, data });

  // Bump the version, old cached data becomes orphaned (expires naturally)
  await redis.setex(`profile:${userId}:version`, 3600, Date.now().toString());

  return updated;
}

On update, bump the version. The old versioned cache entry becomes unreachable (nothing references it anymore) and expires naturally. The next read fetches fresh data. No delete-then-repopulate race because you're never deleting a key that another request is about to write.

This uses more memory (orphaned keys until TTL) and adds a round trip for the version lookup, but eliminates the race condition.

Fix 4: Event-Driven Invalidation

For production systems where data is written from multiple services, application-level invalidation breaks down. You need to hear about writes from the database itself.

PostgreSQL's LISTEN/NOTIFY:

// Listener process
const client = new Client();
await client.connect();
await client.query('LISTEN profile_updated');

client.on('notification', async (msg) => {
  const { userId } = JSON.parse(msg.payload);
  await redis.del(`profile:${userId}`);
});

// In your update function (PostgreSQL trigger or application code)
await db.$executeRaw`
  SELECT pg_notify('profile_updated', ${JSON.stringify({ userId })})
`;

Or publish invalidation events to a message queue (Redis pub/sub, Kafka, SQS) that cache nodes subscribe to. Any service that writes to the database publishes an invalidation event. Cache nodes clear the relevant keys. No relying on application code in the write path to remember to invalidate.

What the Fix Actually Was

For our specific case: we switched cache misses to read from primary, dropped TTL from 3600 to 300, and added an explicit note in code about the replica lag issue so the next developer doesn't "optimize" it away.

async function getUserProfile(userId: string) {
  const cacheKey = `profile:${userId}`;

  const cached = await redis.get(cacheKey);
  if (cached) return JSON.parse(cached);

  // Reads from primary, replica lag would cause stale repopulation
  const profile = await db.primary.user.findUnique({
    where: { id: userId },
    select: { id: true, name: true, bio: true, avatarUrl: true },
  });

  await redis.setex(cacheKey, 300, JSON.stringify(profile)); // 5 min TTL
  return profile;
}

Not the most elegant solution. But it fixed the user's problem, reduced primary DB load significantly compared to no cache, and the 5-minute TTL means any edge case stale data self-heals quickly.

The Actual Lesson

Cache-aside looks simple. It has at least three places where data can become permanently stale:

1. Your invalidation code has a bug
2. Another service writes to the same data without invalidating your cache
3. The cache-miss repopulation reads stale data from a replica

A cache is not a transparent layer. It's a consistency trade-off that you're making explicitly. Design for the failure cases before they become production incidents.

And set a TTL. Always set a TTL.