INFO : Exception: Too many counters: 121 max=120
at org.apache.tez.common.counters.Limits.checkCounters(Limits.java:86)
at org.apache.tez.common.counters.Limits.incrCounters(Limits.java:93)
at org.apache.tez.common.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:76)
at org.apache.tez.common.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:93)
at org.apache.tez.common.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:104)
at org.apache.tez.common.counters.AbstractCounterGroup.aggrAllCounters(AbstractCounterGroup.java:204)
at org.apache.tez.common.counters.AbstractCounters.aggrAllCounters(AbstractCounters.java:372)
at org.apache.tez.dag.app.dag.impl.DAGImpl.aggrTaskCounters(DAGImpl.java:773)
at org.apache.tez.dag.app.dag.impl.DAGImpl.getAllCounters(DAGImpl.java:722)
at org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:938)
at org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:981)
This is an error reported when executing Hive SQL, the engine uses Tez.
The mapreduce.job.counters.max
parameter of the mapred-site.xml
configuration file of Hadoop needs to be modified. The default value is 120
, which needs to be set to a larger value.
<property>
<name>mapreduce.job.counters.max</name>
<value>200</value>
</property>
After adjustment, the error is reported again: Concurrency mode is disabled, not creating a lock manager
The error message is as follows:
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask.
INFO : Completed executing command(queryId=dingli_20200311174547_a0c682f0-af9e-4f13-97c3-f4f076e88621); Time taken: 63.102 seconds
INFO : Concurrency mode is disabled, not creating a lock manager
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask. (state=08S01,code=1)
solution
Modify the hive configuration file, uncomment the configuration exprot HADOOP_HEAPSIZE=1024
cd $HIVE_HOME/conf
cp hive-env.sh.template hive-env.sh
vim hive-env.sh
exprot HADOOP_HEAPSIZE=1024