Caching: Strategies, Eviction and Pitfalls

Every time a product page loads, the same product description is fetched from the database for the thousandth time. A cache keeps a copy of frequently used data in fast storage, usually memory, so most requests never touch the database at all. It is often the cheapest performance win in a design, and also a rich source of subtle bugs.

Why caching works

Real traffic is skewed: a small share of items (popular products, trending posts) gets most of the reads. Memory is far faster than disk or a network call to a database. Combine the two and a cache can cut average latency and shield the database from load. The average latency is the weighted mix of hits and misses. Using illustrative numbers of 1 ms for a cache hit, 20 ms for a database read and a 90 percent hit ratio, the average is 0.9 x 1 + 0.1 x 20 = 2.9 ms, against 20 ms with no cache. The hit ratio is the number to watch: raising it from 90 to 99 percent drops the average to about 1.2 ms in this example.

Caches can live at several levels: in the browser, on a CDN, inside the application process, in a shared cache such as Redis or Memcached, and in the database itself. Closer to the user is faster, but harder to invalidate.

Caching strategies

Cache-aside is the most common default. The flow looks like this:

  App server
     |  1. GET profile:42
     v
  +-------+   hit    returns data immediately
  | Cache |--------->
  +-------+
     | miss
     v  2. SELECT ... WHERE id = 42
  +----------+
  | Database |
  +----------+
     |  3. return row
     v
  App server: 4. SET profile:42 (TTL 5 min)
              5. respond to client

async function getProfile(id: string) {
  const key = "profile:" + id;

  const cached = await cache.get(key);
  if (cached !== null) {
    return JSON.parse(cached); // cache hit
  }

  const row = await db.findProfile(id); // cache miss: go to the source
  if (row) {
    // TTL bounds how stale this entry can get
    await cache.set(key, JSON.stringify(row), { ttlSeconds: 300 });
  }
  return row;
}

async function updateProfile(id: string, changes: object) {
  await db.updateProfile(id, changes);
  await cache.del("profile:" + id); // invalidate so the next read reloads
}

Eviction policies and TTL

Memory is limited, so a full cache must decide what to throw out. A TTL (time to live) expires an entry after a fixed time and bounds staleness. An eviction policy chooses victims when memory is full.

LRU is a classic interview exercise. A Map in JavaScript keeps insertion order, so re-inserting a key on each access moves it to the "most recent" end, and the first key is the least recently used.

class LRUCache<K, V> {
  private map = new Map<K, V>();

  constructor(private capacity: number) {}

  get(key: K): V | undefined {
    if (!this.map.has(key)) return undefined;
    const value = this.map.get(key) as V;
    this.map.delete(key);
    this.map.set(key, value); // move to the most recent position
    return value;
  }

  set(key: K, value: V): void {
    if (this.map.has(key)) {
      this.map.delete(key);
    } else if (this.map.size >= this.capacity) {
      const oldest = this.map.keys().next().value as K;
      this.map.delete(oldest); // evict least recently used
    }
    this.map.set(key, value);
  }
}

const cache = new LRUCache<string, number>(2);
cache.set("a", 1);
cache.set("b", 2);
cache.get("a");     // touches "a", so "b" is now the oldest
cache.set("c", 3);  // evicts "b"
console.log(cache.get("b"), cache.get("a"));

undefined 1

Invalidation and consistency

"There are only two hard things in computer science: cache invalidation and naming things" is a joke with a point. The cache and the database can disagree, and you need a policy for how long that is acceptable. The usual tools are, in order of simplicity: rely on a short TTL; delete the cache entry when the data is written (as in updateProfile above); include a version in the key (profile:42:v7) so changes naturally miss; or publish invalidation events from the database layer.

Be careful with race conditions. If one request reads old data from the database while another updates it and clears the cache, the first request can write the stale value back into the cache after the delete. A TTL is your safety net for such cases, which is a good reason to set one even when you also invalidate explicitly.

Cache stampede and hot keys

A cache stampede (or thundering herd) happens when a popular entry expires and thousands of requests miss at the same moment, all hitting the database to rebuild it. The database can be overwhelmed, which makes the rebuild slower, which makes more requests pile up.

• Request coalescing or locking: let one request rebuild the value while others wait or serve the old value.
• TTL jitter: add a small random amount to each TTL so many keys do not expire together.
• Early refresh: refresh popular entries in the background shortly before they expire.
• Serve stale while revalidating: briefly return the expired value while one worker fetches the new one.

CDN: caching close to the user

A content delivery network is a cache spread across many locations. Static assets such as images, scripts, stylesheets and videos are cached at edge servers near users, which reduces latency and removes load from your origin servers. Use long cache lifetimes with file names that include a content hash (app.3f9c2.js), so a new deploy changes the URL instead of needing invalidation. Dynamic, personalised responses are usually not cacheable at the CDN, or only with care.

Common mistakes

• Caching without a TTL. Symptom: users see old data until someone manually clears the cache, and bugs surface weeks later.
• Forgetting invalidation on writes. Symptom: a user updates their profile and still sees the old one.
• Cache as the only copy. Treating a cache as durable storage. Symptom: data vanishes after a restart or eviction.
• Caching everything. Low-hit-ratio entries waste memory and push out useful ones. Measure hit ratio before and after.
• Ignoring stampedes on popular keys. Symptom: periodic latency spikes exactly when a popular entry expires.

How this shows up in interviews and real projects

Interviewers expect you to say where the cache sits, which strategy you use, what the eviction and expiry policy is, and what happens on a miss or a cache failure. In real projects, the best first step is to measure hit ratio and latency, then cache the few queries that dominate. In the URL shortener worked example, caching is what makes a read-heavy system cheap to run.