During the execution of HiveSQL, an error Exception: Too many counters: 121 max=120 is reported

problem¶

INFO  : Exception: Too many counters: 121 max=120
at org.apache.tez.common.counters.Limits.checkCounters(Limits.java:86)
at org.apache.tez.common.counters.Limits.incrCounters(Limits.java:93)
at org.apache.tez.common.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:76)
at org.apache.tez.common.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:93)
at org.apache.tez.common.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:104)
at org.apache.tez.common.counters.AbstractCounterGroup.aggrAllCounters(AbstractCounterGroup.java:204)
at org.apache.tez.common.counters.AbstractCounters.aggrAllCounters(AbstractCounters.java:372)
at org.apache.tez.dag.app.dag.impl.DAGImpl.aggrTaskCounters(DAGImpl.java:773)
at org.apache.tez.dag.app.dag.impl.DAGImpl.getAllCounters(DAGImpl.java:722)
at org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:938)
at org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:981)

This is an error reported when executing Hive SQL, the engine uses Tez.

solution¶

The mapreduce.job.counters.max parameter of the mapred-site.xml configuration file of Hadoop needs to be modified. The default value is 120, which needs to be set to a larger value.

<property> 
    <name>mapreduce.job.counters.max</name> 
    <value>200</value> 
</property> 

After adjustment, the error is reported again: Concurrency mode is disabled, not creating a lock manager

The error message is as follows:

ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask. 
INFO  : Completed executing command(queryId=dingli_20200311174547_a0c682f0-af9e-4f13-97c3-f4f076e88621); Time taken: 63.102 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask. (state=08S01,code=1)

solution

Modify the hive configuration file, uncomment the configuration exprot HADOOP_HEAPSIZE=1024

cd $HIVE_HOME/conf
cp hive-env.sh.template hive-env.sh
vim hive-env.sh
exprot HADOOP_HEAPSIZE=1024