ERROR queue.BoundedInMemoryExecutor: error producing records0] org.apache.parquet.io.ParquetDecoding

created at 11-24-2021 views: 2

error

ERROR queue.BoundedInMemoryExecutor: error producing records0]
org.apache.parquet.io.ParquetDecodingException: Can not read value at 1 in block 0 in file hdfs://hdp-yl-1:8020/user/testJoin/test_join27/join/default/1d0f7a5b-fcbc-40aa-994d-ada47e3a3257-0_0-59-5054_20211119171950.parquet

Reasons and solution

The reason for the error is that the data type of the field to be written into the form and the destination form field are different.

The solution is to reset the data type of the written data. See the example below.

write_df2 = write_df2.withColumn("superior_emp_id",col("superior_emp_id").cast("string"))
created at:11-24-2021
edited at: 11-24-2021: