Can't save in background: fork: Cannot allocate memory Redis

Using Redis!!! Well a great choice indeed but you might have face a downtime because the Redis server fails miserably and if you see the logs you might get the following log by running:

$ tail -f /var/log/redis.log

Logs

4294:C 03 Apr 16:58:16.114 * RDB: 221 MB of memory used by copy-on-write
944:M 03 Apr 16:58:16.213 * Background saving terminated with success
944:M 03 Apr 17:03:17.024 * 10 changes in 300 seconds. Saving...
944:M 03 Apr 17:03:17.024 # Can't save in background: fork: Cannot allocate memory
944:M 03 Apr 17:03:23.037 * 10 changes in 300 seconds. Saving...
944:M 03 Apr 17:03:23.037 # Can't save in background: fork: Cannot allocate memory
944:M 03 Apr 17:03:29.048 * 10 changes in 300 seconds. Saving...
944:M 03 Apr 17:03:29.048 # Can't save in background: fork: Cannot allocate memory
944:M 03 Apr 17:03:35.061 * 10 changes in 300 seconds. Saving...

This error is caused when the Redis forks a child process to write data to disk and ask for the memory but if the memory is not sufficient enough, it won't be allocated the memory hence after several retries the Redis server shuts down.

If you further observe the logs, you will get the solution to this problem too.

5400:M 03 Apr 18:21:19.894 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.

So as seen above in the logs , operating system's overcommit settings were causing this problem. Following is the explanation, how linux allocates memory.

In traditional way, Linux gives you three options for what happens when a process tries to allocate some more memory, controlled by the value of the vm.overcommit_memory sysctl:

• the kernel gives you the memory unless it thinks you would clearly overcommit the system (mode 0, the default, 'heuristic overcommit').
• the kernel always gives you the memory (mode 1, 'always overcommit').
• the kernel refuses to give you more memory if it would take the committed address space over the commit limit (mode 2, what I call 'strict overcommit').

(Disclaimer: all of this assumes a relatively recent 2.6 kernel.)

These settings control how Linux handles virtual memory limits.

After changing this setting from "0" to "1" Redis started persisting the data immediately and the overall performance increased dramatically.
To do so either open the file /proc/sys/vm/overcommit_memory and remove 0 and put 1 or run the following command.

echo 1 > /proc/sys/vm/overcommit_memory

This might now work as the file may be already in use by the system. So other way to do the same is to add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.


From Redis FAQ

Redis background saving schema relies on the copy-on-write semantic of fork in modern operating systems: Redis forks (creates a child process) that is an exact copy of the parent. The child process dumps the DB on disk and finally exits. In theory the child should use as much memory as the parent being a copy, but actually thanks to the copy-on-write semantic implemented by most modern operating systems the parent and child process will share the common memory pages. A page will be duplicated only when it changes in the child or in the parent. Since in theory all the pages may change while the child process is saving, Linux can't tell in advance how much memory the child will take, so if the overcommit_memory setting is set to zero fork will fail unless there is as much free RAM as required to really duplicate all the parent memory pages, with the result that if you have a Redis dataset of 3 GB and just 2 GB of free memory it will fail.

Setting overcommit_memory to 1 says Linux to relax and perform the fork in a more optimistic allocation fashion, and this is indeed what you want for Redis.