codis: ResponseError: handle response, backend conn failure, bad array len, too long

created at 07-22-2021 views: 8

When using codis, the error message is as follows:

ResponseError: handle response, backend conn failure, bad array len, too long

background

There are more than 500w key data in a hash table. When using the hkeys command, an error ResponseError: handle response, backend conn failure, bad array len, too long will be displayed.

If the number of keys in an ordered set exceeds 180w pieces of data, when using zrange, the error ResponseError: handle response, backend conn failure, bad array len, too long will also be prompted.

problem causes

const (
    MaxBulkBytesLen = 1024 * 1024 * 512
    MaxArrayLen     = 1024 * 1024
)

This is because the codes of the codes limit the number of keys not to exceed 1024*1024, and the maximum data returned at a time does not exceed 1024*1024*500 bytes.

According to the Redis protocol, Codis should not have more than 1024x1024 data in BulkBytesArray.

func (d *Decoder) decodeArray() ([]*Resp, error) {    
    n, err := d.decodeInt()    
    if err != nil {    
        return nil, err    
    }    
    switch {    
    case n < -1:    
        return nil, errors.Trace(ErrBadArrayLen)    
    case n > :    
        return nil, errors.Trace(ErrBadArrayLenTooLong)    
    case n == -1:    
        return nil, nil    
    }    
    array := make([]*Resp, n)    
    for i := range array {    
        r, err := d.decodeResp()    
        if err != nil {    
            return nil, err    
        }    
        array[i] = r    
    }    
    return array, nil    
}

Solution

  1. Split into multiple tables.
  2. Modify the codis source code.
  3. If it is caused by the large amount of data obtained, it is recommended to extract the data in batches.
created at:07-22-2021
edited at: 07-22-2021: