java.lang.IllegalArgumentException: Unknown message type: 10

created at 06-13-2022 views: 59

problem

spark opens ESS go error: java.lang.IllegalArgumentException: Unknown message type: 10

The error log after spark upgrade is as follows

ERROR ShuffleBlockFetcherIterator: Error occurred while fetching host local blocks
java.lang.RuntimeException: java.lang.IllegalArgumentException: Unknown message type: 10

reason and solution

Check the source code and find org.apache.spark.network.shuffle.protocol.BlockTransferMessage The type of has increased a lot;

Need to enable spark.shuffle.useOldFetchProtocol = true to be compatible with the old communication protocol.

created at:06-13-2022
edited at: 06-13-2022: