Why is Redis faster with pipelining?

created at 07-27-2021 views: 6

This is a very detailed question, and most people will say: Because the network overhead is reduced, then look at the following example:

import time

import redis

client = redis.Redis(decode_responses=True)
count = 10000

def no_pipelining():
    for i in range(count):
        client.set("test:nopp:{}".format(i), i, ex=100)

def with_pipelining():
    pp = client.pipeline()

    for i in range(count):
        pp.set("test:withpp:{}".format(i), i, ex=100)


if __name__ == "__main__":
    start = time.time()
    mid = time.time()
    end = time.time()

    print("no_pipelining: {} seconds; with_pipelining: {} seconds".format(mid - start, end - mid))

Why is there such a big difference in execution results?

$ python test.py
no_pipelining: 2.3809118270874023 seconds; with_pipelining: 0.4370129108428955 seconds

Because this is connected to the local redis, the network overhead is very small. Of course, there is still a part of the network overhead impact, but are there other influencing factors besides this? The answer is yes, such as OS process scheduling. When the pipeline is not used, there is a time gap between Redis processing each command. Therefore, the OS is likely to switch the Redis process to sleep and then run other programs while using pipelining It can improve the CPU utilization, and Redis does not have so much idle time. Therefore, this is also one of the important reasons for the faster pipelining speed.

created at:07-27-2021
edited at: 07-27-2021: