训练出现'pickle data was truncated'错误

[love-ll]

您好！在训练时出现'pickle data was truncated'错误，我重新下载了pkl文件还是出现同样的问题，想问一下是什么原因造成的呢？
完整的输出如下：

Namespace(arch='Document-Dewarping-with-Control-Points', batch_size=8, data_path_test=PosixPath('/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/dataset/fiducial1024/png'), data_path_train='/media/alyson/DataDisk1/fiducial1024/fiducial1024/fiducial1024_v1', data_path_validate='/media/alyson/DataDisk1/fiducial1024/fiducial1024/fiducial1024_v1/validate', img_shrink=None, l_rate=0.0002, n_epoch=300, optimizer='adam', output_path=PosixPath('/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/flat'), parallel=['0'], print_freq=60, resume=PosixPath('/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/ICDAR2021/2021-02-03 16:15:55/143/2021-02-03 16_15_55flat_img_by_fiducial_points-fiducial1024_v1.pkl'), schema='train')

------load DilatedResnetForFlatByFiducialPointsS2------

Loading model and optimizer from checkpoint '/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/ICDAR2021/2021-02-03 16:15:55/143/2021-02-03 16_15_55flat_img_by_fiducial_points-fiducial1024_v1.pkl'
Loaded checkpoint '2021-02-03 16_15_55flat_img_by_fiducial_points-fiducial1024_v1.pkl' (epoch 143)

• lambda_loss :1 learning_rate :5e-05
  /home/alyson/anaconda3/envs/wave/lib/python3.8/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='mean' instead.
  warnings.warn(warning.format(ret))
  [144][60/2308] [10.86 18.7846 32.55] [l1:8.5548 l:102.2232 e:0.0000 r:0.0000 s:0.7517] 18.7846 2023-03-07 16:23:33
  [144][120/2308] [11.70 19.4594 28.32] [l1:7.7310 l:117.2133 e:0.0000 r:0.0000 s:0.7067] 19.1220 2023-03-07 16:24:41
  [144][180/2308] [10.56 17.9781 42.29] [l1:6.9421 l:110.2881 e:0.0000 r:0.0000 s:0.7259] 18.7407 2023-03-07 16:25:52
  [144][240/2308] [10.04 18.7990 28.57] [l1:8.0336 l:107.5737 e:0.0000 r:0.0000 s:0.7993] 18.7553 2023-03-07 16:27:02
  [144][300/2308] [11.42 18.5368 35.10] [l1:7.5782 l:109.5079 e:0.0000 r:0.0000 s:0.7803] 18.7116 2023-03-07 16:28:17
  [144][360/2308] [10.29 18.0371 28.19] [l1:7.2705 l:107.5887 e:0.0000 r:0.0000 s:0.7711] 18.5992 2023-03-07 16:29:27
  Traceback (most recent call last):
  File "/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/train.py", line 334, in
  train(args)
  File "/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/train.py", line 143, in train
  for i, (images, labels, segment) in enumerate(trainloader):
  File "/home/alyson/anaconda3/envs/wave/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in next
  data = self._next_data()
  File "/home/alyson/anaconda3/envs/wave/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1183, in _next_data
  return self._process_data(data)
  File "/home/alyson/anaconda3/envs/wave/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
  data.reraise()
  File "/home/alyson/anaconda3/envs/wave/lib/python3.8/site-packages/torch/_utils.py", line 425, in reraise
  raise self.exc_type(msg)
  _pickle.UnpicklingError: Caught UnpicklingError in DataLoader worker process 3.
  Original Traceback (most recent call last):
  File "/home/alyson/anaconda3/envs/wave/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
  data = fetcher.fetch(index)
  File "/home/alyson/anaconda3/envs/wave/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
  data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/alyson/anaconda3/envs/wave/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
  data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/dataloader.py", line 146, in getitem
  perturbed_data = pickle.load(f)
  _pickle.UnpicklingError: pickle data was truncated

[gwxie]

你好，

程序在[360/2308]是正常运行的，可能是数据有问题。建议你可以dataloader.py中pickle处使用try...except查看下是哪个数据出了问题。

[Testhjf]

您好！在训练时出现'pickle data was truncated'错误，我重新下载了pkl文件还是出现同样的问题，想问一下是什么原因造成的呢？ 完整的输出如下：

Namespace(arch='Document-Dewarping-with-Control-Points', batch_size=8, data_path_test=PosixPath('/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/dataset/fiducial1024/png'), data_path_train='/media/alyson/DataDisk1/fiducial1024/fiducial1024/fiducial1024_v1', data_path_validate='/media/alyson/DataDisk1/fiducial1024/fiducial1024/fiducial1024_v1/validate', img_shrink=None, l_rate=0.0002, n_epoch=300, optimizer='adam', output_path=PosixPath('/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/flat'), parallel=['0'], print_freq=60, resume=PosixPath('/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/ICDAR2021/2021-02-03 16:15:55/143/2021-02-03 16_15_55flat_img_by_fiducial_points-fiducial1024_v1.pkl'), schema='train')

------load DilatedResnetForFlatByFiducialPointsS2------

Loading model and optimizer from checkpoint '/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/ICDAR2021/2021-02-03 16:15:55/143/2021-02-03 16_15_55flat_img_by_fiducial_points-fiducial1024_v1.pkl' Loaded checkpoint '2021-02-03 16_15_55flat_img_by_fiducial_points-fiducial1024_v1.pkl' (epoch 143)

• lambda_loss :1 learning_rate :5e-05
  /home/alyson/anaconda3/envs/wave/lib/python3.8/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='mean' instead.
  warnings.warn(warning.format(ret))
  [144][60/2308] [10.86 18.7846 32.55] [l1:8.5548 l:102.2232 e:0.0000 r:0.0000 s:0.7517] 18.7846 2023-03-07 16:23:33
  [144][120/2308] [11.70 19.4594 28.32] [l1:7.7310 l:117.2133 e:0.0000 r:0.0000 s:0.7067] 19.1220 2023-03-07 16:24:41
  [144][180/2308] [10.56 17.9781 42.29] [l1:6.9421 l:110.2881 e:0.0000 r:0.0000 s:0.7259] 18.7407 2023-03-07 16:25:52
  [144][240/2308] [10.04 18.7990 28.57] [l1:8.0336 l:107.5737 e:0.0000 r:0.0000 s:0.7993] 18.7553 2023-03-07 16:27:02
  [144][300/2308] [11.42 18.5368 35.10] [l1:7.5782 l:109.5079 e:0.0000 r:0.0000 s:0.7803] 18.7116 2023-03-07 16:28:17
  [144][360/2308] [10.29 18.0371 28.19] [l1:7.2705 l:107.5887 e:0.0000 r:0.0000 s:0.7711] 18.5992 2023-03-07 16:29:27
  Traceback (most recent call last):
  File "/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/train.py", line 334, in
  train(args)
  File "/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/train.py", line 143, in train
  for i, (images, labels, segment) in enumerate(trainloader):
  File "/home/alyson/anaconda3/envs/wave/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in next
  data = self._next_data()
  File "/home/alyson/anaconda3/envs/wave/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1183, in _next_data
  return self._process_data(data)
  File "/home/alyson/anaconda3/envs/wave/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
  data.reraise()
  File "/home/alyson/anaconda3/envs/wave/lib/python3.8/site-packages/torch/_utils.py", line 425, in reraise
  raise self.exc_type(msg)
  _pickle.UnpicklingError: Caught UnpicklingError in DataLoader worker process 3.
  Original Traceback (most recent call last):
  File "/home/alyson/anaconda3/envs/wave/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
  data = fetcher.fetch(index)
  File "/home/alyson/anaconda3/envs/wave/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
  data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/alyson/anaconda3/envs/wave/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
  data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/alyson/PycharmProjects/Document-Dewarping-with-Control-Points/Source/dataloader.py", line 146, in getitem
  perturbed_data = pickle.load(f)
  _pickle.UnpicklingError: pickle data was truncated

后来训练成功了吗