MyBatis uses Zookeeper to save the configuration of the database, which can be dynamically refreshed
created at 08-05-2021
views: 5
Core key point: Encapsulate a DataSource and rewrite getConnection to achieve
Let's look at it step by step.
Environment:
- Spring Cloud
- MyBatis
Configure the data source in the conventional way of MyBatis: use Spring's Configuration¶
package com.cnscud.cavedemo.fundmain.config;
import com.cnscud.xpower.dbn.SimpleDBNDataSourceFactory;
import org.apache.ibatis.session.SqlSessionFactory;
import org.mybatis.spring.SqlSessionFactoryBean;
import org.mybatis.spring.SqlSessionTemplate;
import org.mybatis.spring.annotation.MapperScan;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.jdbc.DataSourceBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.core.io.support.PathMatchingResourcePatternResolver;
import org.springframework.jdbc.datasource.DataSourceTransactionManager;
import org.springframework.transaction.PlatformTransactionManager;
import javax.sql.DataSource;
/**
* Database Config Multi-data source configuration: the main data source.
*
* @author Felix Zhang 2021-08-02 17:30
* @version 1.0.0
*/
@Configuration
@MapperScan(basePackages = {"com.cnscud.cavedemo.fundmain.dao"},
sqlSessionFactoryRef = "sqlSessionFactoryMainDataSource")
public class MainDataSourceConfig {
//General configuration: Use the configuration in application.yml.
@Primary
@Bean(name = "mainDataSource")
@ConfigurationProperties("spring.datasource.main")
public DataSource mainDataSource() throws Exception {
return DataSourceBuilder.create().build();
}
@Primary
@Bean(name = "sqlSessionFactoryMainDataSource")
public SqlSessionFactory sqlSessionFactoryMainDataSource(@Qualifier("mainDataSource") DataSource mainDataSource) throws Exception {
SqlSessionFactoryBean factoryBean = new SqlSessionFactoryBean();
//org.apache.ibatis.session.Configuration configuration = new org.apache.ibatis.session.Configuration();
//configuration.setMapUnderscoreToCamelCase(true);
//factoryBean.setConfiguration(configuration);
factoryBean.setConfigLocation(new PathMatchingResourcePatternResolver().getResource("classpath:mybatis-config.xml"));
// Use mainDataSource data source, connect to mainDataSource library
factoryBean.setDataSource(mainDataSource);
// The following two sentences are only used for *.xml files, if the entire persistence layer operation does not need to use the xml file (just use annotations to get it), do not add
//Specify the path of entity and mapper xml
//factoryBean.setTypeAliasesPackage("com.cnscud.cavedemo.fundmain.model");
factoryBean.setMapperLocations(new PathMatchingResourcePatternResolver().getResources("classpath:com/cnscud/cavedemo/fundmain/mapper/*.xml"));
return factoryBean.getObject();
}
@Primary
@Bean
public SqlSessionTemplate sqlSessionTemplateMainDataSource(@Qualifier("sqlSessionFactoryMainDataSource") SqlSessionFactory sqlSessionTemplateMainDataSource) throws Exception {
//Use the factory configured in the annotation
return new SqlSessionTemplate(sqlSessionTemplateMainDataSource);
}
@Primary
@Bean
public PlatformTransactionManager mainTransactionManager(@Qualifier("mainDataSource") DataSource prodDataSource) {
return new DataSourceTransactionManager(prodDataSource);
}
}
The key function to get the data source here is mainDataSource, we just need to implement it ourselves:
Because this is a one-time job, we can't modify the data source's pointer, we can only make a fuss inside the DataSource, so we need to implement a DataSource ourselves.
There are many steps, let's take a look at the final result:
The final DataSourceWrapper¶
It completely encapsulates a DataSource, and does not have any DataSource functions:
package com.cnscud.xpower.dbn;
import javax.sql.DataSource;
import java.io.PrintWriter;
import java.sql.Connection;
import java.sql.SQLException;
import java.sql.SQLFeatureNotSupportedException;
import java.util.logging.Logger;
/**
* Datasource wrapper, to facilitate the creation of DataSource dynamically.
*
* @author Felix Zhang 2021-08-05 14:14
* @version 1.0.0
*/
public class DynamicByZookeeperDataSourceWrapper implements DataSource {
protected SimpleDBNConnectionPool simpleDBNConnectionPool;
protected String bizName;
public DynamicByZookeeperDataSourceWrapper(SimpleDBNConnectionPool simpleDBNConnectionPool, String bizName) {
this.simpleDBNConnectionPool = simpleDBNConnectionPool;
this.bizName = bizName;
}
protected DataSource pickDataSource() throws SQLException{
return simpleDBNConnectionPool.getDataSource(bizName);
}
@Override
public Connection getConnection() throws SQLException {
return pickDataSource().getConnection();
}
@Override
public Connection getConnection(String username, String password) throws SQLException {
return pickDataSource().getConnection(username, password);
}
@Override
public <T> T unwrap(Class<T> iface) throws SQLException {
return pickDataSource().unwrap(iface);
}
@Override
public boolean isWrapperFor(Class<?> iface) throws SQLException {
return pickDataSource().isWrapperFor(iface);
}
@Override
public PrintWriter getLogWriter() throws SQLException {
return pickDataSource().getLogWriter();
}
@Override
public void setLogWriter(PrintWriter out) throws SQLException {
pickDataSource().setLogWriter(out);
}
@Override
public void setLoginTimeout(int seconds) throws SQLException {
pickDataSource().setLoginTimeout(seconds);
}
@Override
public int getLoginTimeout() throws SQLException {
return pickDataSource().getLoginTimeout();
}
@Override
public Logger getParentLogger() throws SQLFeatureNotSupportedException {
throw new SQLFeatureNotSupportedException();
}
}
SimpleDBNConnectionPool¶
Support the staging pool of multiple data sources, you can get different database DataSource instances according to the name:
This class is responsible for creating a DataSource and saving it in the Map. It can also monitor Zookeeper changes. Once the changes are detected, it closes the existing DataSource.
package com.cnscud.xpower.dbn;
import com.github.zkclient.IZkDataListener;
import com.zaxxer.hikari.HikariDataSource;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import javax.sql.DataSource;
import java.sql.Connection;
import java.sql.SQLException;
import java.util.HashSet;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import static java.lang.String.format;
/**
* The simple datasource pool.
*
* Store the DataSource of multiple databases according to the name, and will monitor the Zookeeper configuration and dynamically rebuild.
*
* @author adyliu (imxylz@gmail.com)
* @since 2011-7-27
*/
public class SimpleDBNConnectionPool {
final Logger logger = LoggerFactory.getLogger(getClass());
private Map<String, DataSource> instances = new ConcurrentHashMap<>();
private final Set<String> watcherSchema = new HashSet<String>();
public DataSource getInstance(String bizName) {
try {
return findDbInstance(bizName);
}
catch (SQLException e) {
e.printStackTrace();
}
return null;
}
public Connection getConnection(String bizName) throws SQLException {
DataSource ds = getDataSource(bizName);
return ds.getConnection();
}
public DataSource getDataSource(String bizName) throws SQLException {
return findDbInstance(bizName);
}
protected void destroyInstance(final String bizName) {
synchronized (instances) {
DataSource oldInstanceIf = instances.remove(bizName);
logger.warn(format("destoryInstance %s and %s", bizName, oldInstanceIf != null ? "close datasource" : "do nothing"));
if (oldInstanceIf != null) {
closeDataSource(oldInstanceIf);
}
}
}
protected void closeDataSource(DataSource ds) {
if (ds instanceof HikariDataSource) {
try {
((HikariDataSource) ds).close();
}
catch (Exception e) {
logger.error("Close datasource failed. ", e);
}
}
}
private DataSource createInstance(Map<String, String> dbcfg) {
return new SimpleDataSourceBuilder().buildDataSource(dbcfg);
}
private DataSource findDbInstance(final String bizName) throws SQLException {
DataSource ins = instances.get(bizName);
if (ins != null) {
return ins;
}
synchronized (instances) {// Synchronous operation
ins = instances.get(bizName);
if (ins != null) {
return ins;
}
boolean success = false;
try {
Map<String, String> dbcfg = SchemeNodeHelper.getInstance(bizName);
if (dbcfg == null) {
throw new SQLException("No such datasouce: " + bizName);
}
ins = createInstance(dbcfg);
//log.warn("ins put "+ins);
instances.put(bizName, ins);
if (watcherSchema.add(bizName)) {
SchemeNodeHelper.watchInstance(bizName, new IZkDataListener() {
public void handleDataDeleted(String dataPath) throws Exception {
logger.warn(dataPath + " was deleted, so destroy the bizName " + bizName);
destroyInstance(bizName);
}
public void handleDataChange(String dataPath, byte[] data) throws Exception {
logger.warn(dataPath + " was changed, so destroy the bizName " + bizName);
destroyInstance(bizName);
}
});
}
success = true;
}
catch (SQLException e) {
throw e;
}
catch (Throwable t) {
throw new SQLException("cannot build datasource for bizName: " + bizName, t);
}
finally {
if (!success) {
instances.remove(bizName);
}
}
}
return ins;
}
}
The code to actually create the DataSource:¶
package com.cnscud.xpower.dbn;
import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
import org.apache.commons.lang.StringUtils;
import java.util.Map;
/**
* Hikari DataSource.
*
* Thinking: You can use different libraries to create DataSource according to the type in the parameter, such as Druid. (The default is HikariDataSource)
*
*
* @author Felix Zhang 2021-08-05 11:14
* @version 1.0.0
*/
public class SimpleDataSourceBuilder {
public HikariDataSource buildDataSource(Map<String, String> args) {
HikariConfig config = new HikariConfig();
config.setJdbcUrl(getUrl(args));
config.setUsername(args.get("username"));
config.setPassword(args.get("password"));
config.setDriverClassName(getDriverClassName(args));
String maximumPoolSizeKey = "maximum-pool-size";
int maximumPoolSize = 30;
if(StringUtils.isNotEmpty(args.get(maximumPoolSizeKey))){
maximumPoolSize = Integer.parseInt(args.get(maximumPoolSizeKey));
}
config.addDataSourceProperty("cachePrepStmts", "true"); //Whether to customize the configuration, the following two parameters will take effect when it is true
config.addDataSourceProperty("prepStmtCacheSize", maximumPoolSize); //The default connection pool size is 25, and the official recommendation is 250-500
config.addDataSourceProperty("prepStmtCacheSqlLimit", "2048"); //The maximum length of a single sentence is 256 by default, and the official recommendation is 2048
config.addDataSourceProperty("useServerPrepStmts", "true"); //The new version of MySQL supports server-side preparation, enabling it to get a significant performance improvement
config.addDataSourceProperty("useLocalSessionState", "true");
config.addDataSourceProperty("useLocalTransactionState", "true");
config.addDataSourceProperty("rewriteBatchedStatements", "true");
config.addDataSourceProperty("cacheResultSetMetadata", "true");
config.addDataSourceProperty("cacheServerConfiguration", "true");
config.addDataSourceProperty("elideSetAutoCommits", "true");
config.addDataSourceProperty("maintainTimeStats", "false");
config.setMaximumPoolSize(maximumPoolSize); //
config.setMinimumIdle(10);//The minimum number of idle connections, the default is 0
config.setMaxLifetime(600000);//Maximum survival time
config.setConnectionTimeout(30000);//30 seconds timeout
config.setIdleTimeout(60000);
config.setConnectionTestQuery("select 1");
return new HikariDataSource(config);
}
private String getDriverClassName(Map<String, String> args) {
return args.get("driver-class-name");
}
private String getUrl(Map<String, String> args) {
return args.get("jdbc-url") == null ? args.get("url"): args.get("jdbc-url");
}
}
In order to facilitate reading Zookeeper nodes, there is also a SchemeNodeHelper:
Two configuration file formats are supported: json or Properties format:
package com.cnscud.xpower.dbn;
import com.cnscud.xpower.configcenter.ConfigCenter;
import com.cnscud.xpower.utils.Jsons;
import com.github.zkclient.IZkDataListener;
import org.apache.commons.lang.StringUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.io.StringReader;
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;
/**
* Read the database configuration from Zookeeper's /xpower/dbn node.
* The content supports two formats: json or properties format.
*
* The JSON format is as follows:
* {
* "jdbc-url": "jdbc:mysql://127.0.0.1:3306/cavedemo?useUnicode=true&characterEncoding=utf8&useSSL=false&serverTimezone=UTC",
* "username": "dbuser",
* "password": "yourpassword",
* "driver-class-name": "com.mysql.cj.jdbc.Driver"
* }
*
* The Properties format is as follows:
* jdbc-url: jdbc:mysql://127.0.0.1:3306/cavedemo?useUnicode=true&characterEncoding=utf8&useSSL=false&serverTimezone=UTC
* username: dbuser
* password: password
* driver-class-name: com.mysql.cj.jdbc.Driver
*
* @author Felix Zhang
* @since 2021-8-5
*/
public class SchemeNodeHelper {
static final Logger logger = LoggerFactory.getLogger(SchemeNodeHelper.class);
//Two formats are supported: json, properties
public static Map<String, String> getInstance(final String instanceName) throws Exception {
String data = ConfigCenter.getInstance().getDataAsString("/xpower/dbn/" + instanceName);
if(StringUtils.isEmpty(data)){
return null;
}
data = data.trim();
if (data.startsWith("{")) {
//as json
Map<String, String> swap = Jsons.fromJson(data, Map.class);
Map<String, String> result = new HashMap<>();
if (swap != null) {
for (String name : swap.keySet()) {
result.put(name.toLowerCase(), swap.get(name));
}
}
return result;
}
else {
//as properties
Properties props = new Properties();
try {
props.load(new StringReader(data));
}
catch (IOException e) {
logger.error("loading global config failed", e);
}
Map<String, String> result = new HashMap<>();
for (String name : props.stringPropertyNames()) {
result.put(name.toLowerCase(), props.getProperty(name));
}
return result;
}
}
public static void watchInstance(final String bizName, final IZkDataListener listener) {
final String path = "/xpower/dbn/" + bizName;
ConfigCenter.getInstance().subscribeDataChanges(path, listener);
}
}
Practical application¶
Finally, in the MyBatis project, replace the original MainDataSource code as:
/**
* Add @Primary annotation, set the default data source, transaction manager.
* A DataSource that can be dynamically rebuilt is used here. If the Zookeeper configuration changes, it will be rebuilt dynamically.
*/
@Primary
@Bean(name = "mainDataSource")
public DataSource mainDataSource() throws Exception {
return SimpleDBNDataSourceFactory.getInstance().getDataSource("cavedemo");
}
Run the project and find that you can connect to the database, and without restarting the project, you can dynamically modify the database configuration to automatically reconnect.
Project code: github
The ConfigCenter used in it is also in this project, or you can implement it yourself, and you can leave this project.
