Kylin Error:Cannot start job scheduler due to lack of job lock

created at 10-20-2021 views: 7

Kylin starts abnormally

The error log is as follows:

2021-10-18 09:34:23,210 DEBUG [localhost-startStop-1] zookeeper.ZookeeperDistributedLock:101 : 71583@gapuat01 trying to lock /job_engine/global_job_engine_lock
2021-10-18 09:34:23,236 DEBUG [localhost-startStop-1] zookeeper.ZookeeperDistributedLock:167 : 71583@gapuat01 see /job_engine/global_job_engine_lock is already locked
2021-10-18 09:34:23,243 DEBUG [localhost-startStop-1] zookeeper.ZookeeperDistributedLock:117 : 71583@gapuat01 failed to acquire lock at /job_engine/global_job_engine_lock, which is held by 13249@gapuat02
2021-10-18 09:34:23,244 DEBUG [localhost-startStop-1] zookeeper.ZookeeperDistributedLock:182 : 71583@gapuat01 will wait for lock path /job_engine/global_job_engine_lock
2021-10-18 09:34:32,091 DEBUG [localhost-startStop-1] zookeeper.ZookeeperDistributedLock:101 : 71583@gapuat01 trying to lock /job_engine/global_job_engine_lock
2021-10-18 09:34:32,096 DEBUG [localhost-startStop-1] zookeeper.ZookeeperDistributedLock:167 : 71583@gapuat01 see /job_engine/global_job_engine_lock is already locked
2021-10-18 09:34:32,098 DEBUG [localhost-startStop-1] zookeeper.ZookeeperDistributedLock:117 : 71583@gapuat01 failed to acquire lock at /job_engine/global_job_engine_lock, which is held by 13249@gapuat02
2021-10-18 09:34:32,100 WARN  [localhost-startStop-1] support.XmlWebApplicationContext:550 : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'migrationService': Unsatisfied dependency expressed through field 'cubeService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cubeMgmtService': Unsatisfied dependency expressed through field 'jobService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jobService' defined in URL [jar:file:/data/software/apache-kylin-3.1.1-bin/tomcat/webapps/kylin/WEB-INF/lib/kylin-server-base-3.1.1.jar!/org/apache/kylin/rest/service/JobService.class]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Cannot start job scheduler due to lack of job lock
2021-10-18 09:34:32,102 WARN  [metrics-blocking-reservoir-scheduler-0] impl.BlockingReservoir:204 : Interrupted during running
2021-10-18 09:34:32,102 INFO  [metrics-blocking-reservoir-scheduler-0] impl.BlockingReservoir:108 : Will report 0 metrics records
2021-10-18 09:34:32,102 INFO  [metrics-blocking-reservoir-scheduler-0] impl.BlockingReservoir:197 : Reporter finishes reporting metrics.
2021-10-18 09:34:32,110 ERROR [localhost-startStop-1] context.ContextLoader:350 : Context initialization failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'migrationService': Unsatisfied dependency expressed through field 'cubeService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'cubeMgmtService': Unsatisfied dependency expressed through field 'jobService'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jobService' defined in URL [jar:file:/data/software/apache-kylin-3.1.1-bin/tomcat/webapps/kylin/WEB-INF/lib/kylin-server-base-3.1.1.jar!/org/apache/kylin/rest/service/JobService.class]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Cannot start job scheduler due to lack of job lock
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:588)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:87)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1257)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:551)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:757)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5197)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5720)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1016)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:992)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:639)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1127)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:2020)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

problem causes

Kylin can only have one node configured as kylin.server.mode=all, and the node configured as all calls the zookeeper distributed lock exception as follows:

failed to acquire lock at /job_engine/global_job_engine_lock, which is held by 13249@gapuat02

It means that gapuat2 has started a node with kylin.server.mode=all, causing the current node to start reporting an error

solution

Ensure that only one kylin server in the cluster is configured as: kylin.server.mode=all

created at:10-20-2021
edited at: 10-20-2021: