Recently, redis publish and subscribe system is used as the message queue of company project. One server publishes messages, and multiple servers subscribe to obtain messages. During the peak period of system usage, the Redis link of some servers was occasionally disconnected and an error was reported at around 10 a.m.:
Uncaught RedisException: socket error on read socket
After checking the redis log, I found such an error log:
Client id=319669 addr=192.168.1.10:52846 fd=83 name= age=534515 idle=1 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=882 omem=452122 events=rw cmd=subscribe scheduled to be closed ASAP for overcoming of output buffer limits
The error log clearly shows that there is a limit on the output buffer for the link that is about to be closed. The preliminary judgment is that the server that sent the message has generated too many messages. The subscribed server was too late to consume and reached the limit of the system output buffer size. This closed the link. Google found that
redis.conf has the following restrictions.
# The client output buffer limits can be used to force disconnection of clients # that are not reading data from the server fast enough for some reason (a # common reason is that a Pub/Sub client can't consume messages as fast as the # publisher can produce them). # # The limit can be set differently for the three different classes of clients: # # normal -> normal clients including MONITOR clients # slave -> slave clients # pubsub -> clients subscribed to at least one pubsub channel or pattern # # The syntax of every client-output-buffer-limit directive is the following: # # client-output-buffer-limit <class> <hard limit> <soft limit> <soft seconds> # # A client is immediately disconnected once the hard limit is reached, or if # the soft limit is reached and remains reached for the specified number of # seconds (continuously). # So for instance if the hard limit is 32 megabytes and the soft limit is # 16 megabytes / 10 seconds, the client will get disconnected immediately # if the size of the output buffers reach 32 megabytes, but will also get # disconnected if the client reaches 16 megabytes and continuously overcomes # the limit for 10 seconds. # # By default normal clients are not limited because they don't receive data # without asking (in a push way), but just after a request, so only # asynchronous clients may create a scenario where data is requested faster # than it can read. # # Instead there is a default limit for pubsub and slave clients, since # subscribers and slaves receive data in a push fashion. # # Both the hard or the soft limit can be disabled by setting them to zero. client-output-buffer-limit normal 0 0 0 client-output-buffer-limit slave 256mb 64mb 60 client-output-buffer-limit pubsub 32mb 8mb 60
By adjusting the limit of the pubsub cache area, the problem of redis link disconnection caused by the size limit of the client cache area can be solved.