java.lang.IllegalArgumentException: Unknown message type: 10

problem¶

spark opens ESS go error: java.lang.IllegalArgumentException: Unknown message type: 10

The error log after spark upgrade is as follows

ERROR ShuffleBlockFetcherIterator: Error occurred while fetching host local blocks
java.lang.RuntimeException: java.lang.IllegalArgumentException: Unknown message type: 10

reason and solution¶

Check the source code and find org.apache.spark.network.shuffle.protocol.BlockTransferMessage The type of has increased a lot;

Need to enable spark.shuffle.useOldFetchProtocol = true to be compatible with the old communication protocol.