Invalidate the whole Nginx reverse proxy cache in production

This is part 3 of 3 in a series of blog posts:

  1. Boosting Djangos performance with Nginx reverse proxy cache
  2. Manually invalidate URLs cached in Nginx's reverse proxy cache
  3. Invalidate the whole Nginx reverse proxy cache in production

We have quiet some pages in our Nginx reverse proxy cache (over 2.5 million). All these pages are different variations of the same page type. If we make big updates in our HTML we want the new version of the pages to be delivered to our users as soon as possible. In this (quiet rare) case we need to invalidate the Nginx reverse proxy cache as a whole.

The simplest way to accomplish this is to delete everything in the reverse proxy cache directory and restart Nginx. On our production system this is not possible, because deleting the contents of the reverse proxy cache directory takes very long. While the system is deleting files from the cache directory Nginx keeps recreating files in the same directory, so the operation never finishes.

Green and blue Nginx reverse proxy cache directories

Here is our little setup how we dealt with this:

First we have two caches called "blue" and "green". Every cache has its own cache and temp directory:

$ ls /data/
nginx-cache-blue  
nginx-cache-green  
nginx-temp-blue  
nginx-temp-green  

Then we have two configuration files. One for each cache:

Configuration file for the "blue" cache, called nginx_proxy_cache_blue.conf:

proxy_cache_path /data/nginx-cache-blue levels=1:2 max_size=10g inactive=1y keys_zone=nginx_cache:40m;  
proxy_temp_path /data/nginx-temp-blue 1 2;  

Configuratiaon file for the "green" cache, called nginx_proxy_cache_green.conf:

proxy_cache_path /data/nginx-cache-green levels=1:2 max_size=10g inactive=1y keys_zone=nginx_cache:40m;  
proxy_temp_path /data/nginx-temp-green 1 2;  

We have a symlink nginx_proxy_cache_active.conf pointing to the config of the currently used cache.

For example when the "green" cache is active it looks like this:

$ ls -alh nginx_proxy_cache_active.conf
lrwxrwxrwx 1 root root 33 Oct 24 13:11 nginx_proxy_cache_active.conf -> nginx_proxy_cache_green.conf  

In the http section of our nginx.conf we include the symlink like this:

include /etc/nginx/nginx_proxy_cache_active.conf;  

Now Nginx uses he configuration of the "green" cache. If we need to invalidate the whole cache we just change the symlink so it points to the config of the "blue" cache and gracefully restart Nginx.

Now Nginx starts to fill the "blue" cache with the updated pages. We can now delete the "green" cache and it does not matter if it takes days, Nginx is serving the new HTML from the "green" cache.

We keep a simple text file in our file system into which we write "blue" respectively "green" so we know what cache is currently in use.

Keep in mind that when you invalidate the whole cache the disk I/O will increase depending on the load of your system. For us this approach works good. It takes one or two hours until the most popular pages are all cached again and the site's response times is fast again. In these first two hours the response times of the site is a little slower because of all the generation of cache files but this is OK for us.

We you have other solutions to this problem or other ways in handling things with Nginx reverse proxy cache leave me a comment. I am always eager to learn!

I hope you liked the posts about the reverse proxy cache feature of Nginx and you can use some of it in your own projects. Make sure you also check out the other posts in this series!

Further reading

This is a blog post in a series of blog posts about the Nginx reverse proxy feature:

  1. Boosting Djangos performance with Nginx reverse proxy cache
  2. Manually invalidate URLs cached in Nginx's reverse proxy cache
  3. Invalidate the whole Nginx reverse proxy cache in production