ERROR queue.BoundedInMemoryExecutor: error producing records0] org.apache.parquet.io.ParquetDecoding

error¶

ERROR queue.BoundedInMemoryExecutor: error producing records0]
org.apache.parquet.io.ParquetDecodingException: Can not read value at 1 in block 0 in file hdfs://hdp-yl-1:8020/user/testJoin/test_join27/join/default/1d0f7a5b-fcbc-40aa-994d-ada47e3a3257-0_0-59-5054_20211119171950.parquet

Reasons and solution¶

The reason for the error is that the data type of the field to be written into the form and the destination form field are different.

The solution is to reset the data type of the written data. See the example below.

write_df2 = write_df2.withColumn("superior_emp_id",col("superior_emp_id").cast("string"))