Paddlepaddle uses multiple processes to report an error "(External) CUDA error(3), initialization error."

created at 12-08-2021 views: 8

error

When using paddlepaddle to train the model, there is a problem that the model training has ended, but the GPU memory is still occupied, which affects the next training. In order to be able to automatically release the GPU memory after the model training is over, refer to the method of releasing memory by multi-process by Tensorflow. The model training of paddlepaddle can be put into multi-process, so that the GPU resources will be automatically executed after the training process is over. Release.

But sometimes when using multiprocessing to train paddlepaddle model, will get

CUDA error(3), initialization error.

error message.

solution

Refer to the issue discussion of paddlepaddle on github and found that 

all the modules related to paddle are imported in multiprocessing and do not import outside of the multi-process. These modules can run normally, so that the corresponding resources will be automatically after the end of the process. freed.

created at:12-08-2021
edited at: 12-08-2021: