The company's server is down. The slave library of the mysql cluster is deployed on this server. The main library is compiled and installed, and the slave library is installed by docker. Log a startup problem. The problem has been solved, but not the correct way, hope someone can add.
It is found that the server hangs and restarts immediately. All functions are available. When testing the mysql slave library, a problem is found. The connection using the visual tool fails, and the mysql service cannot be connected.
An error is reported when the mysql command line is used to connect to the database inside the container.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysql/mysql.sock' (111)
This file is automatically generated after the database is started, so it is considered that mysql failed to start. However, docker shows that the startup was successful. After entering the container, it has been kicked out of the container. At first, it was thought that other colleagues were operating. Later, it was found that the mysql service in the container had been trying to start the automatic restart of the container.
Because the database cluster has been running smoothly after many power outages, this time I did a random inspection and found the problem.
docker logs name
The following is displayed (the error part is intercepted)
2022-06-06T04:21:07.672350Z 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11 2022-06-06T04:21:07.672375Z 0 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2022-06-06T04:21:07.672377Z 0 [Note] InnoDB: Retrying to lock the first data file 2022-06-06T04:21:08.672532Z 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11 2022-06-06T04:21:08.672577Z 0 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. 2022-06-06T04:21:09.672975Z 0 [ERROR] InnoDB: Unable to lock ./ibdata1 error: 11 2022-06-06T04:22:47.706756Z 0 [Note] InnoDB: Unable to open the first data file 2022-06-06T04:22:47.706778Z 0 [ERROR] InnoDB: Operating system error number 11 in a file operation. 2022-06-06T04:22:47.706803Z 0 [ERROR] InnoDB: Error number 11 means 'Resource temporarily unavailable' 2022-06-06T04:22:47.706811Z 0 [Note] InnoDB: Some operating system error numbers are described at http://dev.mysql.com/doc/refman/5.7/en/operating-system-error-codes.html 2022-06-06T04:22:47.706823Z 0 [ERROR] InnoDB: Cannot open datafile './ibdata1' 2022-06-06T04:22:47.706857Z 0 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data! 2022-06-06T04:22:47.706868Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Cannot open a file 2022-06-06T04:22:48.308050Z 0 [ERROR] Plugin 'InnoDB' init function returned error. 2022-06-06T04:22:48.308098Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 2022-06-06T04:22:48.308110Z 0 [ERROR] Failed to initialize builtin plugins. 2022-06-06T04:22:48.308115Z 0 [ERROR] Aborting 2022-06-06T04:22:48.308124Z 0 [Note] Binlog end 2022-06-06T04:22:48.308238Z 0 [Note] Shutting down plugin 'CSV' 2022-06-06T04:22:48.311617Z 0 [Note] mysqld: Shutdown complete
So far the startup failed, completely
Three solutions given on the Internet
delete the corresponding file
kill the service contending for files
Move the corresponding file
Considering that docker is started, there is no possibility of contention, so this one is directly excluded.
Finally, directly use the
mv command to move the corresponding file, which is equivalent to deleting and backing up, and then copy it back with
cp -a to forcibly unlock it.
In fact, this problem should be checked where the lock occurs. There is an article about the problem of the main library. I thought about restarting the main library, but this is impossible.
At this point, after the copy comes back, the library can be connected and accessed normally.
show slave status \G;
The master-slave state is also normal.