Skip to main content

Caching: The Secret Behind It All

Caching is one of the most fundamental concepts in backend engineering. It's the process of storing copies of data in a temporary storage location (a cache) so that future requests for that data can be served faster.

1. Why Do We Need Caching?​

Caching is primarily used to:

  • Reduce Latency: Accessing data from a cache is much faster than fetching it from the original source (e.g., a database or an external API).
  • Reduce Load: By serving popular data from a cache, you reduce the workload on your primary database or servers.
  • Save Costs: Reducing the number of hits to expensive resources (like third-party APIs) can significantly lower operational costs.
  • Improve Scalability: Caching allows your system to handle more concurrent users by offloading repetitive tasks.

2. Caching at Different Levels​

Caching happens at multiple layers of a modern technical infrastructure:

A. Network Level Caching​

  • CDN (Content Delivery Network): Distributes static assets (images, videos, JS/CSS) to edge nodes geographically closer to the user.
  • DNS Caching: Resolvers cache IP address lookups to avoid repeated queries to root, TLD, and authoritative name servers.

B. Hardware Level Caching​

  • CPU Caches (L1, L2, L3): Extremely fast memory built into the processor to store frequently accessed instructions and data.
  • RAM (Main Memory): Large, volatile memory that stores data currently being used by the operating system and applications.
  • Secondary Storage (SSD/HDD): Non-volatile storage where data persists. Accessing data here is thousands of times slower than RAM.

C. Software Level Caching​

  • In-Memory Databases: Technologies like Redis and Memcached store data entirely in RAM for sub-millisecond retrieval.
  • Browser Caching: Browsers store static resources locally to avoid re-downloading them on every visit.

3. Caching Strategies​

How data is placed in and synchronized with the cache determines the strategy:

Cache-Aside (Lazy Caching)​

  1. The application checks the cache first.
  2. Cache Hit: Returns data immediately.
  3. Cache Miss: Fetches data from the database, stores it in the cache, and then returns it to the user. Best for read-heavy workloads where data doesn't change frequently.

Write-Through Caching​

  1. Data is written to both the cache and the primary database simultaneously. Ensures the cache is always up-to-date, but adds latency to write operations.

4. Eviction Policies​

Since cache memory (RAM) is limited, we need "Eviction Policies" to decide which data to remove when the cache is full:

  • LRU (Least Recently Used): Removes data that hasn't been accessed for the longest time.
  • LFU (Least Frequently Used): Removes data with the lowest access count.
  • TTL (Time To Live): Automatically expires data after a predefined duration (e.g., 1 hour).
  • No Eviction: Returns an error when the memory is full (common in critical data scenarios).

5. Real-World Use Cases​

Caching is used across various domains in backend engineering:

  1. Database Query Results: Caching the output of compute-intensive SQL joins or aggregations.
  2. E-commerce (Amazon): Product details, prices, and inventory are cached to handle millions of concurrent views.
  3. Social Media (Twitter/Facebook): User profile data is cached since it's frequently read but rarely updated.
  4. Session Management: Storing session tokens in Redis to avoid hitting the database for every single authenticated request.
  5. External API Results: Caching data from third-party APIs (like weather or stock prices) to avoid rate limits and reduce costs.
  6. Rate Limiting: Implementing counters in Redis to track and restrict the number of requests from specific IP addresses (HTTP 429 - Too Many Requests).

Summary​

Caching is a powerful tool to make your applications faster and more reliable. However, it introduces complexity, especially around Cache Invalidation ("There are only two hard things in Computer Science: cache invalidation and naming things"). Choosing the right strategy and eviction policy is key to a successful implementation.