MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk

created at 07-22-2021 views: 437

background

When I came back to work on Monday, I encountered an error message when using codis to perform data operations:

MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

Redis is configured to save database snapshots, but it currently cannot be persisted to the hard disk. The command used to modify the collection data cannot be used. Please check the detailed error information in the Redis log.

reason

There are three reasons for this:

  1. Forcibly shutting down the Redis snapshot results in failure to persist.
  2. This error will be reported when the disk is full
  3. This error occurs because BGSAVE failed. During BGSAVE, Redis spawns a child process to save data on disk. This error can be checked from the log for the exact reason why BGSAVE failed (usually /var/log/redis/redis-server.log on a linux computer), but many times BGAVE fails because fork cannot allocate memory. It is also possible that due to a conflict in the optimization of the OS, the fork cannot allocate memory (although the computer has enough available RAM). (In most cases, this is the error).

If you are using Redis in a very write-heavy application, while saving an RDB file on disk or rewriting the AOF log Redis may use up to 2 times the memory normally used. The additional memory used is proportional to the number of memory pages modified by writes during the saving process, so it is often proportional to the number of keys (or aggregate types items) touched during this time. Make sure to size your memory accordingly.

reference: https://redis.io/topics/admin

Brief description:

For example, the server currently has 8g of memory, other processes occupy 1g of memory, redis occupies 5g of memory, and the remaining available memory is 2g.

When redis triggers bgsave, it will trigger Can't save in background: fork: Cannot allocate memory. Because redis will fork a process in bgsave, because the remaining memory of the system is less than 5g, an error is triggered.

check the log file

cd /var/lib/redis_7021
tail -n 10 redis_7021.log

Temporary solution

After running the config set stop-writes-on-bgsave-error no   command, close the configuration item stop-writes-on-bgsave-error to solve the problem.

The effect of this is: stop saving the snapshot, without actually solving the problem.

root@ubuntu# ./redis-cli
127.0.0.1:6379> config set stop-writes-on-bgsave-error no
OK
127.0.0.1:6379> lpush myColour "red"
(integer) 1

second solution

  1. First check the log file of the server where each codis is located, and find the machine with the error
  2. Try to release the memory of the machine. If it is due to insufficient memory, use the save command to save the data first, and then split the data to another server.
  3. Check whether the hard disk space is insufficient.

Summary

  1. For servers using redis, it is recommended that the remaining space be kept at about 45% as much as possible. If this value is exceeded, the server needs to be expanded or the data transferred.
  2. Rdb files cannot be shared between different versions of redis.
created at:07-22-2021
edited at: 07-22-2021: