spark opens ESS go error: java.lang.IllegalArgumentException: Unknown message type: 10
The error log after spark upgrade is as follows
ERROR ShuffleBlockFetcherIterator: Error occurred while fetching host local blocks
java.lang.RuntimeException: java.lang.IllegalArgumentException: Unknown message type: 10
Check the source code and find org.apache.spark.network.shuffle.protocol.BlockTransferMessage
The type of has increased a lot;
Need to enable spark.shuffle.useOldFetchProtocol = true
to be compatible with the old communication protocol.