codis error Failed to start dashboard: store: acquire lock of codis-testX failed

Background¶

Due to the power outage before spring vacation, the shutdown name shutdown was used directly, and the codis was not manually shut down, which caused the failure to start the codis dashboard.

check the error log¶

cat /usr/local/codis/logs/dashboard.log.2019-02-11

log information:

2019/02/11 13:31:43 topom.go:189: [ERROR] store: acquire lock of codis-testX failed
[error]: zk: node already exists
    6   /opt/gowork/src/github.com/CodisLabs/codis/pkg/models/zk/zkclient.go:247
            github.com/CodisLabs/codis/pkg/models/zk.(*Client).create
    5   /opt/gowork/src/github.com/CodisLabs/codis/pkg/models/zk/zkclient.go:196
            github.com/CodisLabs/codis/pkg/models/zk.(*Client).Create.func1
    4   /opt/gowork/src/github.com/CodisLabs/codis/pkg/models/zk/zkclient.go:129
            github.com/CodisLabs/codis/pkg/models/zk.(*Client).shell
    3   /opt/gowork/src/github.com/CodisLabs/codis/pkg/models/zk/zkclient.go:195
            github.com/CodisLabs/codis/pkg/models/zk.(*Client).Create
    2   /opt/gowork/src/github.com/CodisLabs/codis/pkg/models/store.go:119
            github.com/CodisLabs/codis/pkg/models.(*Store).Acquire
    1   /opt/gowork/src/github.com/CodisLabs/codis/pkg/topom/topom.go:188
            github.com/CodisLabs/codis/pkg/topom.(*Topom).Start
    0   /opt/gowork/src/github.com/CodisLabs/codis/cmd/dashboard/main.go:169
            main.main
        ... ...
[stack]: 
    1   /opt/gowork/src/github.com/CodisLabs/codis/pkg/topom/topom.go:189
            github.com/CodisLabs/codis/pkg/topom.(*Topom).Start
    0   /opt/gowork/src/github.com/CodisLabs/codis/cmd/dashboard/main.go:169
            main.main
        ... ...

Need to delete zookeeper nodes¶

Specific code example:

1. Confirm that the codis-dashboard process has exited (very important)

codis-admin --dashboard=192.168.3.190:18080 --shutdown

2. Then run codis-admin to delete LOCK:

codis-admin --remove-lock --product=codis_demo --zookeeper=127.0.0.1:2181

#### result
2019/07/25 10:51:11 zkclient.go:23: [INFO] zookeeper - zkclient setup new connection to 127.0.0.1:2181
2019/07/25 10:51:11 zkclient.go:23: [INFO] zookeeper - Connected to 127.0.0.1:2181
2019/07/25 10:51:11 zkclient.go:23: [INFO] zookeeper - Authenticated: id=72370115218046981, timeout=40000
2019/07/25 10:51:11 zkclient.go:23: [INFO] zookeeper - Re-submitting `0` credentials after reconnect
2019/07/25 10:51:11 zkclient.go:23: [INFO] zookeeper - Recv loop terminated: err=EOF
2019/07/25 10:51:11 zkclient.go:23: [INFO] zookeeper - Send loop terminated: err=<nil>

• codis_demo is the project name of codis
• 127.0.0.1:2181 the corresponding  ip address of zookeeper

at last:

Restart codis-dashboard