forked from CGCL-codes/naturalcc
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathpython.log
245 lines (245 loc) · 59.7 KB
/
python.log
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
nohup: ignoring input
[2021-03-22 09:43:09] INFO >> Load arguments in /home/wanyao/yang/naturalcc-dev/run/completion/seqrnn/config/raw_py150/python.yml (train.py:301, cli_main())
[2021-03-22 09:43:09] INFO >> {'criterion': 'completion_cross_entropy', 'optimizer': 'torch_adam', 'lr_scheduler': 'fixed', 'tokenizer': None, 'bpe': None, 'common': {'no_progress_bar': 0, 'log_interval': 500, 'log_format': 'simple', 'tensorboard_logdir': '', 'memory_efficient_fp16': 0, 'fp16_no_flatten_grads': 0, 'fp16_init_scale': 128, 'fp16_scale_window': None, 'fp16_scale_tolerance': 0.0, 'min_loss_scale': 0.0001, 'threshold_loss_scale': None, 'empty_cache_freq': 0, 'task': 'completion', 'seed': 666, 'cpu': 0, 'fp16': 0, 'fp16_opt_level': '01', 'server_ip': '', 'server_port': ''}, 'dataset': {'num_workers': 3, 'skip_invalid_size_inputs_valid_test': 0, 'max_tokens': 100000.0, 'max_sentences': 32, 'required_batch_size_multiple': 1, 'dataset_impl': 'mmap', 'train_subset': 'train', 'valid_subset': 'test', 'validate_interval': 1, 'fixed_validation_seed': None, 'disable_validation': 0, 'max_tokens_valid': None, 'max_sentences_valid': 64, 'curriculum': 5, 'gen_subset': 'test', 'num_shards': 1, 'shard_id': 0}, 'distributed_training': {'distributed_world_size': 4, 'distributed_rank': 0, 'distributed_backend': 'nccl', 'distributed_init_method': None, 'distributed_port': -1, 'device_id': 0, 'distributed_no_spawn': 0, 'ddp_backend': 'no_c10d', 'bucket_cap_mb': 25, 'fix_batches_to_gpus': None, 'find_unused_parameters': 0, 'fast_stat_sync': 0, 'broadcast_buffers': 0, 'global_sync_iter': 50, 'warmup_iterations': 500, 'local_rank': -1}, 'task': {'data': '/home/wanyao/.ncc/raw_py150/completion/data-mmap', 'target_lang': 'code_tokens', 'ext': 'code_tokens.ext', 'code_types': ['attr', 'num', 'name', 'param'], 'max_target_positions': 500, 'add_bos_token': 0, 'eval_bleu': 0, 'eval_bleu_detok': 'space', 'eval_bleu_detok_args': None, 'eval_tokenized_bleu': 0, 'eval_bleu_remove_bpe': None, 'eval_bleu_args': None, 'eval_bleu_print_samples': 0, 'eval_mrr': 1}, 'model': {'arch': 'completion_seqrnn', 'dropout': 0.5, 'decoder_embed_dim': 300, 'decoder_embed_path': None, 'decoder_freeze_embed': None, 'decoder_hidden_size': 300, 'decoder_layers': 1, 'max_target_positions': 500}, 'optimization': {'max_epoch': 100, 'max_update': 0, 'clip_norm': 25, 'update_freq': [1], 'lrs': [0.001], 'min_lr': -1, 'use_bmuf': 0, 'force_anneal': None, 'warmup_updates': 0, 'end_learning_rate': 0.0, 'power': 1.0, 'total_num_update': 1000000, 'sentence_avg': None, 'adam': {'adam_betas': '(0.9, 0.999)', 'adam_eps': 1e-08, 'weight_decay': 0.0, 'use_old_adam': 0}, 'weight_decay': 0.0, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 5, 'max_steps': -1, 'warmup_steps': 0, 'gradient_accumulation_steps': 1}, 'checkpoint': {'save_dir': '/home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints', 'restore_file': 'checkpoint_best.pt', 'reset_dataloader': None, 'reset_lr_scheduler': None, 'reset_meters': None, 'reset_optimizer': None, 'optimizer_overrides': '{}', 'save_interval': 1, 'save_interval_updates': 0, 'keep_interval_updates': 0, 'keep_last_epochs': -1, 'keep_best_checkpoints': -1, 'no_save': 0, 'no_epoch_checkpoints': 1, 'no_last_checkpoints': 0, 'no_save_optimizer_state': None, 'best_checkpoint_metric': 'mrr', 'maximize_best_checkpoint_metric': 1, 'patience': 5, 'should_continue': 0, 'model_name_or_path': None, 'cache_dir': None, 'logging_steps': 500, 'save_steps': 2000, 'save_total_limit': 2, 'overwrite_output_dir': 0, 'overwrite_cache': 0}, 'eval': {'path': '/home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt', 'model_overrides': '{}', 'checkpoint_suffix': '', 'max_sentences_eval': 64}} (train.py:303, cli_main())
[2021-03-22 09:43:11] INFO >> distributed init (rank 1): tcp://localhost:17604 (distributed_utils.py:89, distributed_init())
[2021-03-22 09:43:11] INFO >> distributed init (rank 2): tcp://localhost:17604 (distributed_utils.py:89, distributed_init())
[2021-03-22 09:43:11] INFO >> distributed init (rank 3): tcp://localhost:17604 (distributed_utils.py:89, distributed_init())
[2021-03-22 09:43:11] INFO >> distributed init (rank 0): tcp://localhost:17604 (distributed_utils.py:89, distributed_init())
[2021-03-22 09:43:18] INFO >> initialized host node13 as rank 1 (distributed_utils.py:98, distributed_init())
[2021-03-22 09:43:18] INFO >> initialized host node13 as rank 2 (distributed_utils.py:98, distributed_init())
[2021-03-22 09:43:18] INFO >> initialized host node13 as rank 3 (distributed_utils.py:98, distributed_init())
[2021-03-22 09:43:18] INFO >> initialized host node13 as rank 0 (distributed_utils.py:98, distributed_init())
[2021-03-22 09:43:18] INFO >> {'criterion': 'completion_cross_entropy', 'optimizer': 'torch_adam', 'lr_scheduler': 'fixed', 'tokenizer': None, 'bpe': None, 'common': {'no_progress_bar': 0, 'log_interval': 500, 'log_format': 'simple', 'tensorboard_logdir': '', 'memory_efficient_fp16': 0, 'fp16_no_flatten_grads': 0, 'fp16_init_scale': 128, 'fp16_scale_window': None, 'fp16_scale_tolerance': 0.0, 'min_loss_scale': 0.0001, 'threshold_loss_scale': None, 'empty_cache_freq': 0, 'task': 'completion', 'seed': 666, 'cpu': 0, 'fp16': 0, 'fp16_opt_level': '01', 'server_ip': '', 'server_port': ''}, 'dataset': {'num_workers': 3, 'skip_invalid_size_inputs_valid_test': 0, 'max_tokens': 100000.0, 'max_sentences': 32, 'required_batch_size_multiple': 1, 'dataset_impl': 'mmap', 'train_subset': 'train', 'valid_subset': 'test', 'validate_interval': 1, 'fixed_validation_seed': None, 'disable_validation': 0, 'max_tokens_valid': None, 'max_sentences_valid': 64, 'curriculum': 5, 'gen_subset': 'test', 'num_shards': 1, 'shard_id': 0}, 'distributed_training': {'distributed_world_size': 4, 'distributed_rank': 0, 'distributed_backend': 'nccl', 'distributed_init_method': 'tcp://localhost:17604', 'distributed_port': -1, 'device_id': 0, 'distributed_no_spawn': 0, 'ddp_backend': 'no_c10d', 'bucket_cap_mb': 25, 'fix_batches_to_gpus': None, 'find_unused_parameters': 0, 'fast_stat_sync': 0, 'broadcast_buffers': 0, 'global_sync_iter': 50, 'warmup_iterations': 500, 'local_rank': -1}, 'task': {'data': '/home/wanyao/.ncc/raw_py150/completion/data-mmap', 'target_lang': 'code_tokens', 'ext': 'code_tokens.ext', 'code_types': ['attr', 'num', 'name', 'param'], 'max_target_positions': 500, 'add_bos_token': 0, 'eval_bleu': 0, 'eval_bleu_detok': 'space', 'eval_bleu_detok_args': None, 'eval_tokenized_bleu': 0, 'eval_bleu_remove_bpe': None, 'eval_bleu_args': None, 'eval_bleu_print_samples': 0, 'eval_mrr': 1}, 'model': {'arch': 'completion_seqrnn', 'dropout': 0.5, 'decoder_embed_dim': 300, 'decoder_embed_path': None, 'decoder_freeze_embed': None, 'decoder_hidden_size': 300, 'decoder_layers': 1, 'max_target_positions': 500}, 'optimization': {'max_epoch': 100, 'max_update': 0, 'clip_norm': 25, 'update_freq': [1], 'lrs': [0.001], 'min_lr': -1, 'use_bmuf': 0, 'force_anneal': None, 'warmup_updates': 0, 'end_learning_rate': 0.0, 'power': 1.0, 'total_num_update': 1000000, 'sentence_avg': None, 'adam': {'adam_betas': '(0.9, 0.999)', 'adam_eps': 1e-08, 'weight_decay': 0.0, 'use_old_adam': 0}, 'weight_decay': 0.0, 'adam_epsilon': 1e-08, 'max_grad_norm': 1.0, 'num_train_epochs': 5, 'max_steps': -1, 'warmup_steps': 0, 'gradient_accumulation_steps': 1}, 'checkpoint': {'save_dir': '/home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints', 'restore_file': 'checkpoint_best.pt', 'reset_dataloader': None, 'reset_lr_scheduler': None, 'reset_meters': None, 'reset_optimizer': None, 'optimizer_overrides': '{}', 'save_interval': 1, 'save_interval_updates': 0, 'keep_interval_updates': 0, 'keep_last_epochs': -1, 'keep_best_checkpoints': -1, 'no_save': 0, 'no_epoch_checkpoints': 1, 'no_last_checkpoints': 0, 'no_save_optimizer_state': None, 'best_checkpoint_metric': 'mrr', 'maximize_best_checkpoint_metric': 1, 'patience': 5, 'should_continue': 0, 'model_name_or_path': None, 'cache_dir': None, 'logging_steps': 500, 'save_steps': 2000, 'save_total_limit': 2, 'overwrite_output_dir': 0, 'overwrite_cache': 0}, 'eval': {'path': '/home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt', 'model_overrides': '{}', 'checkpoint_suffix': '', 'max_sentences_eval': 64}} (train.py:212, single_main())
[2021-03-22 09:43:18] INFO >> [code_tokens] dictionary: 50000 types (completion.py:97, setup_task())
[2021-03-22 09:43:18] INFO >> [code_tokens] dictionary: 19 types (completion.py:101, setup_task())
[2021-03-22 09:43:18] INFO >> loaded 122521 examples from: /home/wanyao/.ncc/raw_py150/completion/data-mmap/test.code_tokens (completion.py:44, load_token_dataset())
[2021-03-22 09:43:18] INFO >> loaded 122521 examples from: /home/wanyao/.ncc/raw_py150/completion/data-mmap/test.code_types (completion.py:58, load_token_dataset())
/home/wanyao/anaconda3/envs/py37-1.7/lib/python3.7/site-packages/torch/nn/modules/rnn.py:61: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.5 and num_layers=1
"num_layers={}".format(dropout, num_layers))
[2021-03-22 09:43:19] INFO >> SeqRNNModel(
(decoder): LSTMDecoder(
(embed_tokens): Embedding(50000, 300, padding_idx=0)
(rnn): LSTM(300, 300, batch_first=True, dropout=0.5)
(fc_out): Linear(in_features=300, out_features=50000, bias=True)
)
) (train.py:223, single_main())
[2021-03-22 09:43:19] INFO >> model completion_seqrnn, criterion CompletionCrossEntropyCriterion (train.py:224, single_main())
[2021-03-22 09:43:19] INFO >> num. model params: 30772400 (num. trained: 30772400) (train.py:227, single_main())
[2021-03-22 09:43:19] INFO >> training on 4 GPUs (train.py:232, single_main())
[2021-03-22 09:43:19] INFO >> max tokens per GPU = 100000.0 and max sentences per GPU = 32 (train.py:235, single_main())
[2021-03-22 09:43:19] INFO >> no existing checkpoint found checkpoint_best.pt (ncc_trainer.py:269, load_checkpoint())
[2021-03-22 09:43:19] INFO >> loading train data for epoch 1 (ncc_trainer.py:283, get_train_iterator())
[2021-03-22 09:43:19] INFO >> loaded 253013 examples from: /home/wanyao/.ncc/raw_py150/completion/data-mmap/train.code_tokens (completion.py:44, load_token_dataset())
[2021-03-22 09:43:19] INFO >> loaded 253013 examples from: /home/wanyao/.ncc/raw_py150/completion/data-mmap/train.code_types (completion.py:58, load_token_dataset())
[2021-03-22 09:43:20] INFO >> NOTE: your device may support faster training with fp16 (ncc_trainer.py:154, _setup_optimizer())
/home/wanyao/anaconda3/envs/py37-1.7/lib/python3.7/site-packages/torch/nn/modules/rnn.py:61: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.5 and num_layers=1
"num_layers={}".format(dropout, num_layers))
/home/wanyao/anaconda3/envs/py37-1.7/lib/python3.7/site-packages/torch/nn/modules/rnn.py:61: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.5 and num_layers=1
"num_layers={}".format(dropout, num_layers))
/home/wanyao/anaconda3/envs/py37-1.7/lib/python3.7/site-packages/torch/nn/modules/rnn.py:61: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.5 and num_layers=1
"num_layers={}".format(dropout, num_layers))
/home/wanyao/yang/naturalcc-dev/ncc/utils/utils.py:575: UserWarning: amp_C fused kernels unavailable, disabling multi_tensor_l2norm; you may get better performance by installing NVIDIA's apex library
"amp_C fused kernels unavailable, disabling multi_tensor_l2norm; "
[2021-03-22 09:46:01] INFO >> epoch 001: 500 / 1973 loss=5.388, accuracy=0, mrr=0, ppl=41.88, wps=101759, ups=3.28, wpb=31045.4, bsz=128, num_updates=500, lr=0.001, gnorm=0.417, clip=0, train_wall=152, wall=162 (progress_bar.py:262, log())
[2021-03-22 09:48:33] INFO >> epoch 001: 1000 / 1973 loss=3.942, accuracy=0, mrr=0, ppl=15.37, wps=101825, ups=3.28, wpb=31055.9, bsz=128, num_updates=1000, lr=0.001, gnorm=0.32, clip=0, train_wall=151, wall=314 (progress_bar.py:262, log())
[2021-03-22 09:51:06] INFO >> epoch 001: 1500 / 1973 loss=3.601, accuracy=0, mrr=0, ppl=12.13, wps=101194, ups=3.27, wpb=30979.9, bsz=128, num_updates=1500, lr=0.001, gnorm=0.304, clip=0, train_wall=152, wall=467 (progress_bar.py:262, log())
[2021-03-22 09:53:31] INFO >> epoch 001 | loss 4.085 | accuracy 0 | mrr 0 | ppl 16.97 | wps 101470 | ups 3.27 | wpb 31027.9 | bsz 128 | num_updates 1973 | lr 0.001 | gnorm 0.329 | clip 0 | train_wall 599 | wall 613 (progress_bar.py:269, print())
/home/wanyao/yang/naturalcc-dev/ncc/utils/utils.py:575: UserWarning: amp_C fused kernels unavailable, disabling multi_tensor_l2norm; you may get better performance by installing NVIDIA's apex library
"amp_C fused kernels unavailable, disabling multi_tensor_l2norm; "
/home/wanyao/yang/naturalcc-dev/ncc/utils/utils.py:575: UserWarning: amp_C fused kernels unavailable, disabling multi_tensor_l2norm; you may get better performance by installing NVIDIA's apex library
"amp_C fused kernels unavailable, disabling multi_tensor_l2norm; "
/home/wanyao/yang/naturalcc-dev/ncc/utils/utils.py:575: UserWarning: amp_C fused kernels unavailable, disabling multi_tensor_l2norm; you may get better performance by installing NVIDIA's apex library
"amp_C fused kernels unavailable, disabling multi_tensor_l2norm; "
[2021-03-22 09:56:20] INFO >> epoch 001 | valid on 'test' subset | loss 3.183 | accuracy 0.512493 | mrr 0.590042 | ppl 9.08 | wps 183206 | wpb 61759.1 | bsz 255.8 | num_updates 1973 (progress_bar.py:269, print())
[2021-03-22 09:56:22] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 1 @ 1973 updates, score 0.590042) (writing took 2.668079 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 09:56:38] INFO >> epoch 002: 27 / 1973 loss=3.364, accuracy=0, mrr=0, ppl=10.3, wps=46812.3, ups=1.51, wpb=31027.8, bsz=127.9, num_updates=2000, lr=0.001, gnorm=0.272, clip=0, train_wall=152, wall=799 (progress_bar.py:262, log())
[2021-03-22 09:59:10] INFO >> epoch 002: 527 / 1973 loss=3.173, accuracy=0, mrr=0, ppl=9.02, wps=101810, ups=3.28, wpb=31033.1, bsz=128, num_updates=2500, lr=0.001, gnorm=0.263, clip=0, train_wall=151, wall=951 (progress_bar.py:262, log())
[2021-03-22 10:01:42] INFO >> epoch 002: 1027 / 1973 loss=3.001, accuracy=0, mrr=0, ppl=8, wps=101820, ups=3.28, wpb=31058.5, bsz=128, num_updates=3000, lr=0.001, gnorm=0.249, clip=0, train_wall=151, wall=1104 (progress_bar.py:262, log())
[2021-03-22 10:04:15] INFO >> epoch 002: 1527 / 1973 loss=2.856, accuracy=0, mrr=0, ppl=7.24, wps=101840, ups=3.28, wpb=31005, bsz=128, num_updates=3500, lr=0.001, gnorm=0.242, clip=0, train_wall=151, wall=1256 (progress_bar.py:262, log())
[2021-03-22 10:06:31] INFO >> epoch 002 | loss 2.954 | accuracy 0 | mrr 0 | ppl 7.75 | wps 78492.7 | ups 2.53 | wpb 31027.9 | bsz 128 | num_updates 3946 | lr 0.001 | gnorm 0.249 | clip 0 | train_wall 597 | wall 1393 (progress_bar.py:269, print())
[2021-03-22 10:09:17] INFO >> epoch 002 | valid on 'test' subset | loss 2.703 | accuracy 0.545249 | mrr 0.623178 | ppl 6.51 | wps 186234 | wpb 61759.1 | bsz 255.8 | num_updates 3946 | best_mrr 0.623178 (progress_bar.py:269, print())
[2021-03-22 10:09:22] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 2 @ 3946 updates, score 0.623178) (writing took 5.267884 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 10:09:45] INFO >> epoch 003: 54 / 1973 loss=2.742, accuracy=0, mrr=0, ppl=6.69, wps=46961.6, ups=1.51, wpb=31024, bsz=127.9, num_updates=4000, lr=0.001, gnorm=0.236, clip=0, train_wall=151, wall=1586 (progress_bar.py:262, log())
[2021-03-22 10:12:17] INFO >> epoch 003: 554 / 1973 loss=2.639, accuracy=0, mrr=0, ppl=6.23, wps=101750, ups=3.28, wpb=31024.6, bsz=128, num_updates=4500, lr=0.001, gnorm=0.235, clip=0, train_wall=151, wall=1739 (progress_bar.py:262, log())
[2021-03-22 10:14:51] INFO >> epoch 003: 1054 / 1973 loss=2.542, accuracy=0, mrr=0, ppl=5.83, wps=101458, ups=3.27, wpb=31063.1, bsz=128, num_updates=5000, lr=0.001, gnorm=0.222, clip=0, train_wall=152, wall=1892 (progress_bar.py:262, log())
[2021-03-22 10:17:23] INFO >> epoch 003: 1554 / 1973 loss=2.46, accuracy=0, mrr=0, ppl=5.5, wps=101464, ups=3.27, wpb=31007, bsz=128, num_updates=5500, lr=0.001, gnorm=0.216, clip=0, train_wall=152, wall=2045 (progress_bar.py:262, log())
[2021-03-22 10:19:32] INFO >> epoch 003 | loss 2.52 | accuracy 0 | mrr 0 | ppl 5.74 | wps 78430.6 | ups 2.53 | wpb 31027.9 | bsz 128 | num_updates 5919 | lr 0.001 | gnorm 0.223 | clip 0 | train_wall 598 | wall 2173 (progress_bar.py:269, print())
[2021-03-22 10:22:17] INFO >> epoch 003 | valid on 'test' subset | loss 2.467 | accuracy 0.561206 | mrr 0.639571 | ppl 5.53 | wps 186694 | wpb 61759.1 | bsz 255.8 | num_updates 5919 | best_mrr 0.639571 (progress_bar.py:269, print())
[2021-03-22 10:22:22] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 3 @ 5919 updates, score 0.639571) (writing took 5.288450 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 10:22:53] INFO >> epoch 004: 81 / 1973 loss=2.397, accuracy=0, mrr=0, ppl=5.27, wps=46985.7, ups=1.52, wpb=31012, bsz=127.9, num_updates=6000, lr=0.001, gnorm=0.215, clip=0, train_wall=152, wall=2375 (progress_bar.py:262, log())
[2021-03-22 10:25:26] INFO >> epoch 004: 581 / 1973 loss=2.337, accuracy=0, mrr=0, ppl=5.05, wps=101923, ups=3.28, wpb=31032.9, bsz=128, num_updates=6500, lr=0.001, gnorm=0.211, clip=0, train_wall=151, wall=2527 (progress_bar.py:262, log())
[2021-03-22 10:27:58] INFO >> epoch 004: 1081 / 1973 loss=2.275, accuracy=0, mrr=0, ppl=4.84, wps=101697, ups=3.28, wpb=31043.8, bsz=128, num_updates=7000, lr=0.001, gnorm=0.201, clip=0, train_wall=151, wall=2680 (progress_bar.py:262, log())
[2021-03-22 10:30:31] INFO >> epoch 004: 1581 / 1973 loss=2.223, accuracy=0, mrr=0, ppl=4.67, wps=101296, ups=3.27, wpb=31015.2, bsz=128, num_updates=7500, lr=0.001, gnorm=0.201, clip=0, train_wall=152, wall=2833 (progress_bar.py:262, log())
[2021-03-22 10:32:32] INFO >> epoch 004 | loss 2.264 | accuracy 0 | mrr 0 | ppl 4.8 | wps 78442.9 | ups 2.53 | wpb 31027.9 | bsz 128 | num_updates 7892 | lr 0.001 | gnorm 0.205 | clip 0 | train_wall 599 | wall 2954 (progress_bar.py:269, print())
[2021-03-22 10:35:17] INFO >> epoch 004 | valid on 'test' subset | loss 2.334 | accuracy 0.570586 | mrr 0.648997 | ppl 5.04 | wps 186626 | wpb 61759.1 | bsz 255.8 | num_updates 7892 | best_mrr 0.648997 (progress_bar.py:269, print())
[2021-03-22 10:35:22] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 4 @ 7892 updates, score 0.648997) (writing took 5.092288 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 10:36:02] INFO >> epoch 005: 108 / 1973 loss=2.184, accuracy=0, mrr=0, ppl=4.54, wps=47009.3, ups=1.51, wpb=31059.7, bsz=127.9, num_updates=8000, lr=0.001, gnorm=0.204, clip=0, train_wall=152, wall=3163 (progress_bar.py:262, log())
[2021-03-22 10:38:34] INFO >> epoch 005: 608 / 1973 loss=2.143, accuracy=0, mrr=0, ppl=4.42, wps=101788, ups=3.28, wpb=31011.3, bsz=128, num_updates=8500, lr=0.001, gnorm=0.198, clip=0, train_wall=151, wall=3315 (progress_bar.py:262, log())
[2021-03-22 10:41:07] INFO >> epoch 005: 1108 / 1973 loss=2.103, accuracy=0, mrr=0, ppl=4.3, wps=101579, ups=3.28, wpb=31006.4, bsz=128, num_updates=9000, lr=0.001, gnorm=0.191, clip=0, train_wall=151, wall=3468 (progress_bar.py:262, log())
[2021-03-22 10:43:40] INFO >> epoch 005: 1608 / 1973 loss=2.064, accuracy=0, mrr=0, ppl=4.18, wps=101486, ups=3.27, wpb=31027.6, bsz=128, num_updates=9500, lr=0.001, gnorm=0.189, clip=0, train_wall=152, wall=3621 (progress_bar.py:262, log())
[2021-03-22 10:45:32] INFO >> epoch 005 | loss 2.097 | accuracy 0 | mrr 0 | ppl 4.28 | wps 78557.3 | ups 2.53 | wpb 31027.9 | bsz 128 | num_updates 9865 | lr 0.001 | gnorm 0.196 | clip 0 | train_wall 598 | wall 3733 (progress_bar.py:269, print())
[2021-03-22 10:48:16] INFO >> epoch 005 | valid on 'test' subset | loss 2.251 | accuracy 0.576982 | mrr 0.655244 | ppl 4.76 | wps 186988 | wpb 61759.1 | bsz 255.8 | num_updates 9865 | best_mrr 0.655244 (progress_bar.py:269, print())
[2021-03-22 10:48:21] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 5 @ 9865 updates, score 0.655244) (writing took 4.905102 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 10:49:10] INFO >> epoch 006: 135 / 1973 loss=2.03, accuracy=0, mrr=0, ppl=4.08, wps=47059.7, ups=1.51, wpb=31063.5, bsz=127.9, num_updates=10000, lr=0.001, gnorm=0.198, clip=0, train_wall=152, wall=3951 (progress_bar.py:262, log())
[2021-03-22 10:51:42] INFO >> epoch 006: 635 / 1973 loss=1.99, accuracy=0, mrr=0, ppl=3.97, wps=101358, ups=3.27, wpb=30967.3, bsz=128, num_updates=10500, lr=0.001, gnorm=0.177, clip=0, train_wall=152, wall=4104 (progress_bar.py:262, log())
[2021-03-22 10:54:15] INFO >> epoch 006: 1135 / 1973 loss=1.977, accuracy=0, mrr=0, ppl=3.94, wps=101510, ups=3.27, wpb=31016.4, bsz=128, num_updates=11000, lr=0.001, gnorm=0.178, clip=0, train_wall=152, wall=4256 (progress_bar.py:262, log())
[2021-03-22 10:56:48] INFO >> epoch 006: 1635 / 1973 loss=1.965, accuracy=0, mrr=0, ppl=3.9, wps=101150, ups=3.26, wpb=31016.4, bsz=128, num_updates=11500, lr=0.001, gnorm=0.175, clip=0, train_wall=152, wall=4410 (progress_bar.py:262, log())
[2021-03-22 10:58:33] INFO >> epoch 006 | loss 1.975 | accuracy 0 | mrr 0 | ppl 3.93 | wps 78332.9 | ups 2.52 | wpb 31027.9 | bsz 128 | num_updates 11838 | lr 0.001 | gnorm 0.177 | clip 0 | train_wall 600 | wall 4514 (progress_bar.py:269, print())
[2021-03-22 11:01:19] INFO >> epoch 006 | valid on 'test' subset | loss 2.186 | accuracy 0.582753 | mrr 0.660643 | ppl 4.55 | wps 185940 | wpb 61759.1 | bsz 255.8 | num_updates 11838 | best_mrr 0.660643 (progress_bar.py:269, print())
[2021-03-22 11:01:24] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 6 @ 11838 updates, score 0.660643) (writing took 5.121347 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 11:02:20] INFO >> epoch 007: 162 / 1973 loss=1.938, accuracy=0, mrr=0, ppl=3.83, wps=46918.5, ups=1.51, wpb=31118.4, bsz=127.9, num_updates=12000, lr=0.001, gnorm=0.176, clip=0, train_wall=153, wall=4741 (progress_bar.py:262, log())
[2021-03-22 11:04:53] INFO >> epoch 007: 662 / 1973 loss=1.881, accuracy=0, mrr=0, ppl=3.68, wps=101540, ups=3.27, wpb=31041.1, bsz=128, num_updates=12500, lr=0.001, gnorm=0.171, clip=0, train_wall=152, wall=4894 (progress_bar.py:262, log())
[2021-03-22 11:07:25] INFO >> epoch 007: 1162 / 1973 loss=1.888, accuracy=0, mrr=0, ppl=3.7, wps=102142, ups=3.29, wpb=31063, bsz=128, num_updates=13000, lr=0.001, gnorm=0.173, clip=0, train_wall=151, wall=5046 (progress_bar.py:262, log())
[2021-03-22 11:09:58] INFO >> epoch 007: 1662 / 1973 loss=1.88, accuracy=0, mrr=0, ppl=3.68, wps=101275, ups=3.26, wpb=31028.8, bsz=128, num_updates=13500, lr=0.001, gnorm=0.171, clip=0, train_wall=152, wall=5199 (progress_bar.py:262, log())
[2021-03-22 11:11:34] INFO >> epoch 007 | loss 1.883 | accuracy 0 | mrr 0 | ppl 3.69 | wps 78409.6 | ups 2.53 | wpb 31027.9 | bsz 128 | num_updates 13811 | lr 0.001 | gnorm 0.172 | clip 0 | train_wall 598 | wall 5295 (progress_bar.py:269, print())
[2021-03-22 11:14:19] INFO >> epoch 007 | valid on 'test' subset | loss 2.145 | accuracy 0.586878 | mrr 0.66444 | ppl 4.42 | wps 186643 | wpb 61759.1 | bsz 255.8 | num_updates 13811 | best_mrr 0.66444 (progress_bar.py:269, print())
[2021-03-22 11:14:23] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 7 @ 13811 updates, score 0.66444) (writing took 4.660470 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 11:15:28] INFO >> epoch 008: 189 / 1973 loss=1.854, accuracy=0, mrr=0, ppl=3.61, wps=46974.8, ups=1.52, wpb=30996.4, bsz=127.9, num_updates=14000, lr=0.001, gnorm=0.17, clip=0, train_wall=152, wall=5529 (progress_bar.py:262, log())
[2021-03-22 11:18:01] INFO >> epoch 008: 689 / 1973 loss=1.807, accuracy=0, mrr=0, ppl=3.5, wps=101656, ups=3.28, wpb=31031.8, bsz=128, num_updates=14500, lr=0.001, gnorm=0.17, clip=0, train_wall=151, wall=5682 (progress_bar.py:262, log())
[2021-03-22 11:20:34] INFO >> epoch 008: 1189 / 1973 loss=1.818, accuracy=0, mrr=0, ppl=3.53, wps=101344, ups=3.26, wpb=31075.3, bsz=128, num_updates=15000, lr=0.001, gnorm=0.169, clip=0, train_wall=152, wall=5835 (progress_bar.py:262, log())
[2021-03-22 11:23:06] INFO >> epoch 008: 1689 / 1973 loss=1.813, accuracy=0, mrr=0, ppl=3.51, wps=101511, ups=3.28, wpb=30953.2, bsz=128, num_updates=15500, lr=0.001, gnorm=0.166, clip=0, train_wall=151, wall=5988 (progress_bar.py:262, log())
[2021-03-22 11:24:35] INFO >> epoch 008 | loss 1.811 | accuracy 0 | mrr 0 | ppl 3.51 | wps 78397.4 | ups 2.53 | wpb 31027.9 | bsz 128 | num_updates 15784 | lr 0.001 | gnorm 0.168 | clip 0 | train_wall 599 | wall 6076 (progress_bar.py:269, print())
[2021-03-22 11:27:20] INFO >> epoch 008 | valid on 'test' subset | loss 2.118 | accuracy 0.589337 | mrr 0.666709 | ppl 4.34 | wps 186482 | wpb 61759.1 | bsz 255.8 | num_updates 15784 | best_mrr 0.666709 (progress_bar.py:269, print())
[2021-03-22 11:27:24] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 8 @ 15784 updates, score 0.666709) (writing took 4.330357 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 11:28:37] INFO >> epoch 009: 216 / 1973 loss=1.776, accuracy=0, mrr=0, ppl=3.42, wps=46873.2, ups=1.51, wpb=30992, bsz=127.9, num_updates=16000, lr=0.001, gnorm=0.162, clip=0, train_wall=153, wall=6318 (progress_bar.py:262, log())
[2021-03-22 11:31:11] INFO >> epoch 009: 716 / 1973 loss=1.749, accuracy=0, mrr=0, ppl=3.36, wps=100871, ups=3.26, wpb=30987.2, bsz=128, num_updates=16500, lr=0.001, gnorm=0.167, clip=0, train_wall=152, wall=6472 (progress_bar.py:262, log())
[2021-03-22 11:33:44] INFO >> epoch 009: 1216 / 1973 loss=1.759, accuracy=0, mrr=0, ppl=3.38, wps=101659, ups=3.27, wpb=31122.5, bsz=128, num_updates=17000, lr=0.001, gnorm=0.164, clip=0, train_wall=152, wall=6625 (progress_bar.py:262, log())
[2021-03-22 11:36:17] INFO >> epoch 009: 1716 / 1973 loss=1.755, accuracy=0, mrr=0, ppl=3.38, wps=101299, ups=3.26, wpb=31033.8, bsz=128, num_updates=17500, lr=0.001, gnorm=0.165, clip=0, train_wall=152, wall=6778 (progress_bar.py:262, log())
[2021-03-22 11:37:36] INFO >> epoch 009 | loss 1.752 | accuracy 0 | mrr 0 | ppl 3.37 | wps 78358.4 | ups 2.53 | wpb 31027.9 | bsz 128 | num_updates 17757 | lr 0.001 | gnorm 0.164 | clip 0 | train_wall 600 | wall 6857 (progress_bar.py:269, print())
[2021-03-22 11:40:21] INFO >> epoch 009 | valid on 'test' subset | loss 2.097 | accuracy 0.591373 | mrr 0.668585 | ppl 4.28 | wps 186724 | wpb 61759.1 | bsz 255.8 | num_updates 17757 | best_mrr 0.668585 (progress_bar.py:269, print())
[2021-03-22 11:40:26] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 9 @ 17757 updates, score 0.668585) (writing took 4.979073 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 11:41:47] INFO >> epoch 010: 243 / 1973 loss=1.721, accuracy=0, mrr=0, ppl=3.3, wps=46827.8, ups=1.51, wpb=30936.6, bsz=127.9, num_updates=18000, lr=0.001, gnorm=0.162, clip=0, train_wall=152, wall=7109 (progress_bar.py:262, log())
[2021-03-22 11:44:21] INFO >> epoch 010: 743 / 1973 loss=1.701, accuracy=0, mrr=0, ppl=3.25, wps=101134, ups=3.26, wpb=31057.3, bsz=128, num_updates=18500, lr=0.001, gnorm=0.159, clip=0, train_wall=152, wall=7262 (progress_bar.py:262, log())
[2021-03-22 11:46:54] INFO >> epoch 010: 1243 / 1973 loss=1.705, accuracy=0, mrr=0, ppl=3.26, wps=101272, ups=3.27, wpb=30993.2, bsz=128, num_updates=19000, lr=0.001, gnorm=0.161, clip=0, train_wall=152, wall=7415 (progress_bar.py:262, log())
[2021-03-22 11:49:27] INFO >> epoch 010: 1743 / 1973 loss=1.708, accuracy=0, mrr=0, ppl=3.27, wps=101459, ups=3.27, wpb=31029.1, bsz=128, num_updates=19500, lr=0.001, gnorm=0.16, clip=0, train_wall=152, wall=7568 (progress_bar.py:262, log())
[2021-03-22 11:50:38] INFO >> epoch 010 | loss 1.703 | accuracy 0 | mrr 0 | ppl 3.26 | wps 78280.3 | ups 2.52 | wpb 31027.9 | bsz 128 | num_updates 19730 | lr 0.001 | gnorm 0.16 | clip 0 | train_wall 600 | wall 7639 (progress_bar.py:269, print())
[2021-03-22 11:53:23] INFO >> epoch 010 | valid on 'test' subset | loss 2.079 | accuracy 0.593327 | mrr 0.670363 | ppl 4.23 | wps 186528 | wpb 61759.1 | bsz 255.8 | num_updates 19730 | best_mrr 0.670363 (progress_bar.py:269, print())
[2021-03-22 11:53:28] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 10 @ 19730 updates, score 0.670363) (writing took 5.020858 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 11:54:57] INFO >> epoch 011: 270 / 1973 loss=1.671, accuracy=0, mrr=0, ppl=3.18, wps=47096.3, ups=1.51, wpb=31110.9, bsz=127.9, num_updates=20000, lr=0.001, gnorm=0.156, clip=0, train_wall=152, wall=7898 (progress_bar.py:262, log())
[2021-03-22 11:57:30] INFO >> epoch 011: 770 / 1973 loss=1.653, accuracy=0, mrr=0, ppl=3.15, wps=101745, ups=3.27, wpb=31142.6, bsz=128, num_updates=20500, lr=0.001, gnorm=0.157, clip=0, train_wall=152, wall=8051 (progress_bar.py:262, log())
[2021-03-22 12:00:04] INFO >> epoch 011: 1270 / 1973 loss=1.663, accuracy=0, mrr=0, ppl=3.17, wps=100670, ups=3.25, wpb=31012.3, bsz=128, num_updates=21000, lr=0.001, gnorm=0.159, clip=0, train_wall=153, wall=8205 (progress_bar.py:262, log())
[2021-03-22 12:02:38] INFO >> epoch 011: 1770 / 1973 loss=1.671, accuracy=0, mrr=0, ppl=3.19, wps=100716, ups=3.25, wpb=30997.8, bsz=128, num_updates=21500, lr=0.001, gnorm=0.157, clip=0, train_wall=153, wall=8359 (progress_bar.py:262, log())
[2021-03-22 12:03:41] INFO >> epoch 011 | loss 1.661 | accuracy 0 | mrr 0 | ppl 3.16 | wps 78220.3 | ups 2.52 | wpb 31027.9 | bsz 128 | num_updates 21703 | lr 0.001 | gnorm 0.158 | clip 0 | train_wall 601 | wall 8422 (progress_bar.py:269, print())
[2021-03-22 12:06:26] INFO >> epoch 011 | valid on 'test' subset | loss 2.071 | accuracy 0.594694 | mrr 0.671541 | ppl 4.2 | wps 186336 | wpb 61759.1 | bsz 255.8 | num_updates 21703 | best_mrr 0.671541 (progress_bar.py:269, print())
[2021-03-22 12:06:31] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 11 @ 21703 updates, score 0.671541) (writing took 5.085563 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 12:08:08] INFO >> epoch 012: 297 / 1973 loss=1.631, accuracy=0, mrr=0, ppl=3.1, wps=46828.1, ups=1.51, wpb=30914.9, bsz=127.9, num_updates=22000, lr=0.001, gnorm=0.157, clip=0, train_wall=152, wall=8689 (progress_bar.py:262, log())
[2021-03-22 12:10:41] INFO >> epoch 012: 797 / 1973 loss=1.621, accuracy=0, mrr=0, ppl=3.08, wps=101288, ups=3.27, wpb=31000.6, bsz=128, num_updates=22500, lr=0.001, gnorm=0.155, clip=0, train_wall=152, wall=8842 (progress_bar.py:262, log())
[2021-03-22 12:13:15] INFO >> epoch 012: 1297 / 1973 loss=1.628, accuracy=0, mrr=0, ppl=3.09, wps=101078, ups=3.25, wpb=31116.2, bsz=128, num_updates=23000, lr=0.001, gnorm=0.157, clip=0, train_wall=153, wall=8996 (progress_bar.py:262, log())
[2021-03-22 12:15:48] INFO >> epoch 012: 1797 / 1973 loss=1.633, accuracy=0, mrr=0, ppl=3.1, wps=101226, ups=3.26, wpb=31022.2, bsz=128, num_updates=23500, lr=0.001, gnorm=0.155, clip=0, train_wall=152, wall=9150 (progress_bar.py:262, log())
[2021-03-22 12:16:43] INFO >> epoch 012 | loss 1.624 | accuracy 0 | mrr 0 | ppl 3.08 | wps 78256.3 | ups 2.52 | wpb 31027.9 | bsz 128 | num_updates 23676 | lr 0.001 | gnorm 0.156 | clip 0 | train_wall 600 | wall 9204 (progress_bar.py:269, print())
[2021-03-22 12:19:28] INFO >> epoch 012 | valid on 'test' subset | loss 2.064 | accuracy 0.595652 | mrr 0.672385 | ppl 4.18 | wps 186355 | wpb 61759.1 | bsz 255.8 | num_updates 23676 | best_mrr 0.672385 (progress_bar.py:269, print())
[2021-03-22 12:19:33] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 12 @ 23676 updates, score 0.672385) (writing took 4.937424 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 12:21:19] INFO >> epoch 013: 324 / 1973 loss=1.588, accuracy=0, mrr=0, ppl=3.01, wps=46847.8, ups=1.51, wpb=30969.7, bsz=127.9, num_updates=24000, lr=0.001, gnorm=0.153, clip=0, train_wall=152, wall=9480 (progress_bar.py:262, log())
[2021-03-22 12:23:52] INFO >> epoch 013: 824 / 1973 loss=1.59, accuracy=0, mrr=0, ppl=3.01, wps=101495, ups=3.27, wpb=31060.3, bsz=128, num_updates=24500, lr=0.001, gnorm=0.153, clip=0, train_wall=152, wall=9633 (progress_bar.py:262, log())
[2021-03-22 12:26:25] INFO >> epoch 013: 1324 / 1973 loss=1.599, accuracy=0, mrr=0, ppl=3.03, wps=101290, ups=3.26, wpb=31030.1, bsz=128, num_updates=25000, lr=0.001, gnorm=0.154, clip=0, train_wall=152, wall=9786 (progress_bar.py:262, log())
[2021-03-22 12:28:58] INFO >> epoch 013: 1824 / 1973 loss=1.599, accuracy=0, mrr=0, ppl=3.03, wps=101439, ups=3.27, wpb=31012.9, bsz=128, num_updates=25500, lr=0.001, gnorm=0.154, clip=0, train_wall=152, wall=9939 (progress_bar.py:262, log())
[2021-03-22 12:29:44] INFO >> epoch 013 | loss 1.591 | accuracy 0 | mrr 0 | ppl 3.01 | wps 78365.9 | ups 2.53 | wpb 31027.9 | bsz 128 | num_updates 25649 | lr 0.001 | gnorm 0.153 | clip 0 | train_wall 600 | wall 9985 (progress_bar.py:269, print())
[2021-03-22 12:32:30] INFO >> epoch 013 | valid on 'test' subset | loss 2.06 | accuracy 0.596726 | mrr 0.673244 | ppl 4.17 | wps 186010 | wpb 61759.1 | bsz 255.8 | num_updates 25649 | best_mrr 0.673244 (progress_bar.py:269, print())
[2021-03-22 12:32:35] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 13 @ 25649 updates, score 0.673244) (writing took 5.137585 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 12:34:29] INFO >> epoch 014: 351 / 1973 loss=1.559, accuracy=0, mrr=0, ppl=2.95, wps=46824.3, ups=1.51, wpb=30998.8, bsz=127.9, num_updates=26000, lr=0.001, gnorm=0.151, clip=0, train_wall=152, wall=10270 (progress_bar.py:262, log())
[2021-03-22 12:37:02] INFO >> epoch 014: 851 / 1973 loss=1.559, accuracy=0, mrr=0, ppl=2.95, wps=101222, ups=3.27, wpb=30958.9, bsz=128, num_updates=26500, lr=0.001, gnorm=0.155, clip=0, train_wall=152, wall=10423 (progress_bar.py:262, log())
[2021-03-22 12:39:35] INFO >> epoch 014: 1351 / 1973 loss=1.566, accuracy=0, mrr=0, ppl=2.96, wps=101729, ups=3.27, wpb=31118.2, bsz=128, num_updates=27000, lr=0.001, gnorm=0.153, clip=0, train_wall=152, wall=10576 (progress_bar.py:262, log())
[2021-03-22 12:42:08] INFO >> epoch 014: 1851 / 1973 loss=1.574, accuracy=0, mrr=0, ppl=2.98, wps=101374, ups=3.27, wpb=31003.1, bsz=128, num_updates=27500, lr=0.001, gnorm=0.152, clip=0, train_wall=152, wall=10729 (progress_bar.py:262, log())
[2021-03-22 12:42:46] INFO >> epoch 014 | loss 1.563 | accuracy 0 | mrr 0 | ppl 2.95 | wps 78324.2 | ups 2.52 | wpb 31027.9 | bsz 128 | num_updates 27622 | lr 0.001 | gnorm 0.153 | clip 0 | train_wall 599 | wall 10767 (progress_bar.py:269, print())
[2021-03-22 12:45:31] INFO >> epoch 014 | valid on 'test' subset | loss 2.06 | accuracy 0.596893 | mrr 0.673513 | ppl 4.17 | wps 186495 | wpb 61759.1 | bsz 255.8 | num_updates 27622 | best_mrr 0.673513 (progress_bar.py:269, print())
[2021-03-22 12:45:36] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 14 @ 27622 updates, score 0.673513) (writing took 5.134854 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 12:47:38] INFO >> epoch 015: 378 / 1973 loss=1.533, accuracy=0, mrr=0, ppl=2.89, wps=46997.8, ups=1.51, wpb=31042.5, bsz=127.9, num_updates=28000, lr=0.001, gnorm=0.155, clip=0, train_wall=152, wall=11059 (progress_bar.py:262, log())
[2021-03-22 12:50:11] INFO >> epoch 015: 878 / 1973 loss=1.529, accuracy=0, mrr=0, ppl=2.89, wps=101245, ups=3.27, wpb=30995.8, bsz=128, num_updates=28500, lr=0.001, gnorm=0.15, clip=0, train_wall=152, wall=11212 (progress_bar.py:262, log())
[2021-03-22 12:52:44] INFO >> epoch 015: 1378 / 1973 loss=1.545, accuracy=0, mrr=0, ppl=2.92, wps=101445, ups=3.26, wpb=31104, bsz=128, num_updates=29000, lr=0.001, gnorm=0.151, clip=0, train_wall=152, wall=11366 (progress_bar.py:262, log())
[2021-03-22 12:55:18] INFO >> epoch 015: 1878 / 1973 loss=1.552, accuracy=0, mrr=0, ppl=2.93, wps=100913, ups=3.26, wpb=30988.5, bsz=128, num_updates=29500, lr=0.001, gnorm=0.153, clip=0, train_wall=152, wall=11519 (progress_bar.py:262, log())
[2021-03-22 12:55:48] INFO >> epoch 015 | loss 1.537 | accuracy 0 | mrr 0 | ppl 2.9 | wps 78273.9 | ups 2.52 | wpb 31027.9 | bsz 128 | num_updates 29595 | lr 0.001 | gnorm 0.152 | clip 0 | train_wall 600 | wall 11549 (progress_bar.py:269, print())
[2021-03-22 12:58:33] INFO >> epoch 015 | valid on 'test' subset | loss 2.059 | accuracy 0.597552 | mrr 0.673971 | ppl 4.17 | wps 186696 | wpb 61759.1 | bsz 255.8 | num_updates 29595 | best_mrr 0.673971 (progress_bar.py:269, print())
[2021-03-22 12:58:38] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 15 @ 29595 updates, score 0.673971) (writing took 4.700968 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 13:00:48] INFO >> epoch 016: 405 / 1973 loss=1.503, accuracy=0, mrr=0, ppl=2.83, wps=47103.5, ups=1.51, wpb=31116.3, bsz=127.9, num_updates=30000, lr=0.001, gnorm=0.146, clip=0, train_wall=152, wall=11849 (progress_bar.py:262, log())
[2021-03-22 13:03:21] INFO >> epoch 016: 905 / 1973 loss=1.509, accuracy=0, mrr=0, ppl=2.85, wps=101197, ups=3.27, wpb=30987.6, bsz=128, num_updates=30500, lr=0.001, gnorm=0.149, clip=0, train_wall=152, wall=12003 (progress_bar.py:262, log())
[2021-03-22 13:05:55] INFO >> epoch 016: 1405 / 1973 loss=1.518, accuracy=0, mrr=0, ppl=2.86, wps=101043, ups=3.26, wpb=31032.7, bsz=128, num_updates=31000, lr=0.001, gnorm=0.15, clip=0, train_wall=152, wall=12156 (progress_bar.py:262, log())
[2021-03-22 13:08:28] INFO >> epoch 016: 1905 / 1973 loss=1.527, accuracy=0, mrr=0, ppl=2.88, wps=101381, ups=3.27, wpb=31017.1, bsz=128, num_updates=31500, lr=0.001, gnorm=0.15, clip=0, train_wall=152, wall=12309 (progress_bar.py:262, log())
[2021-03-22 13:08:49] INFO >> epoch 016 | loss 1.513 | accuracy 0 | mrr 0 | ppl 2.85 | wps 78348.8 | ups 2.53 | wpb 31027.9 | bsz 128 | num_updates 31568 | lr 0.001 | gnorm 0.149 | clip 0 | train_wall 600 | wall 12330 (progress_bar.py:269, print())
[2021-03-22 13:11:34] INFO >> epoch 016 | valid on 'test' subset | loss 2.059 | accuracy 0.597431 | mrr 0.673927 | ppl 4.17 | wps 186592 | wpb 61759.1 | bsz 255.8 | num_updates 31568 | best_mrr 0.673971 (progress_bar.py:269, print())
[2021-03-22 13:11:37] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_last.pt (epoch 16 @ 31568 updates, score 0.673927) (writing took 2.774717 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 13:13:57] INFO >> epoch 017: 432 / 1973 loss=1.481, accuracy=0, mrr=0, ppl=2.79, wps=47190.8, ups=1.52, wpb=31034.3, bsz=127.9, num_updates=32000, lr=0.001, gnorm=0.152, clip=0, train_wall=152, wall=12638 (progress_bar.py:262, log())
[2021-03-22 13:16:30] INFO >> epoch 017: 932 / 1973 loss=1.486, accuracy=0, mrr=0, ppl=2.8, wps=101045, ups=3.26, wpb=31040, bsz=128, num_updates=32500, lr=0.001, gnorm=0.148, clip=0, train_wall=152, wall=12792 (progress_bar.py:262, log())
[2021-03-22 13:19:03] INFO >> epoch 017: 1432 / 1973 loss=1.499, accuracy=0, mrr=0, ppl=2.83, wps=101408, ups=3.27, wpb=31007.9, bsz=128, num_updates=33000, lr=0.001, gnorm=0.148, clip=0, train_wall=152, wall=12944 (progress_bar.py:262, log())
[2021-03-22 13:21:36] INFO >> epoch 017: 1932 / 1973 loss=1.505, accuracy=0, mrr=0, ppl=2.84, wps=101186, ups=3.26, wpb=30993.8, bsz=128, num_updates=33500, lr=0.001, gnorm=0.148, clip=0, train_wall=152, wall=13098 (progress_bar.py:262, log())
[2021-03-22 13:21:49] INFO >> epoch 017 | loss 1.492 | accuracy 0 | mrr 0 | ppl 2.81 | wps 78467 | ups 2.53 | wpb 31027.9 | bsz 128 | num_updates 33541 | lr 0.001 | gnorm 0.149 | clip 0 | train_wall 600 | wall 13111 (progress_bar.py:269, print())
[2021-03-22 13:24:34] INFO >> epoch 017 | valid on 'test' subset | loss 2.062 | accuracy 0.598135 | mrr 0.674499 | ppl 4.17 | wps 186440 | wpb 61759.1 | bsz 255.8 | num_updates 33541 | best_mrr 0.674499 (progress_bar.py:269, print())
[2021-03-22 13:24:39] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 17 @ 33541 updates, score 0.674499) (writing took 4.813974 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 13:27:06] INFO >> epoch 018: 459 / 1973 loss=1.455, accuracy=0, mrr=0, ppl=2.74, wps=46912.8, ups=1.52, wpb=30963.7, bsz=127.9, num_updates=34000, lr=0.001, gnorm=0.145, clip=0, train_wall=152, wall=13428 (progress_bar.py:262, log())
[2021-03-22 13:29:40] INFO >> epoch 018: 959 / 1973 loss=1.468, accuracy=0, mrr=0, ppl=2.77, wps=101396, ups=3.26, wpb=31098.5, bsz=128, num_updates=34500, lr=0.001, gnorm=0.148, clip=0, train_wall=152, wall=13581 (progress_bar.py:262, log())
[2021-03-22 13:32:13] INFO >> epoch 018: 1459 / 1973 loss=1.484, accuracy=0, mrr=0, ppl=2.8, wps=101283, ups=3.27, wpb=30998.7, bsz=128, num_updates=35000, lr=0.001, gnorm=0.149, clip=0, train_wall=152, wall=13734 (progress_bar.py:262, log())
[2021-03-22 13:34:46] INFO >> epoch 018: 1959 / 1973 loss=1.487, accuracy=0, mrr=0, ppl=2.8, wps=101372, ups=3.27, wpb=31038.6, bsz=128, num_updates=35500, lr=0.001, gnorm=0.147, clip=0, train_wall=152, wall=13887 (progress_bar.py:262, log())
[2021-03-22 13:34:50] INFO >> epoch 018 | loss 1.472 | accuracy 0 | mrr 0 | ppl 2.77 | wps 78364.3 | ups 2.53 | wpb 31027.9 | bsz 128 | num_updates 35514 | lr 0.001 | gnorm 0.147 | clip 0 | train_wall 599 | wall 13892 (progress_bar.py:269, print())
[2021-03-22 13:37:36] INFO >> epoch 018 | valid on 'test' subset | loss 2.062 | accuracy 0.598184 | mrr 0.674573 | ppl 4.18 | wps 186367 | wpb 61759.1 | bsz 255.8 | num_updates 35514 | best_mrr 0.674573 (progress_bar.py:269, print())
[2021-03-22 13:37:41] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 18 @ 35514 updates, score 0.674573) (writing took 4.888789 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 13:40:16] INFO >> epoch 019: 486 / 1973 loss=1.432, accuracy=0, mrr=0, ppl=2.7, wps=47043.7, ups=1.51, wpb=31057.6, bsz=127.9, num_updates=36000, lr=0.001, gnorm=0.142, clip=0, train_wall=152, wall=14217 (progress_bar.py:262, log())
[2021-03-22 13:42:49] INFO >> epoch 019: 986 / 1973 loss=1.449, accuracy=0, mrr=0, ppl=2.73, wps=101313, ups=3.26, wpb=31062.6, bsz=128, num_updates=36500, lr=0.001, gnorm=0.147, clip=0, train_wall=152, wall=14370 (progress_bar.py:262, log())
[2021-03-22 13:45:23] INFO >> epoch 019: 1486 / 1973 loss=1.464, accuracy=0, mrr=0, ppl=2.76, wps=100626, ups=3.25, wpb=30922.8, bsz=128, num_updates=37000, lr=0.001, gnorm=0.148, clip=0, train_wall=152, wall=14524 (progress_bar.py:262, log())
[2021-03-22 13:47:53] INFO >> epoch 019 | loss 1.454 | accuracy 0 | mrr 0 | ppl 2.74 | wps 78275.3 | ups 2.52 | wpb 31027.9 | bsz 128 | num_updates 37487 | lr 0.001 | gnorm 0.146 | clip 0 | train_wall 600 | wall 14674 (progress_bar.py:269, print())
[2021-03-22 13:50:38] INFO >> epoch 019 | valid on 'test' subset | loss 2.067 | accuracy 0.598369 | mrr 0.674613 | ppl 4.19 | wps 185946 | wpb 61759.1 | bsz 255.8 | num_updates 37487 | best_mrr 0.674613 (progress_bar.py:269, print())
[2021-03-22 13:50:43] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 19 @ 37487 updates, score 0.674613) (writing took 5.131452 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 13:50:54] INFO >> epoch 020: 13 / 1973 loss=1.472, accuracy=0, mrr=0, ppl=2.77, wps=46910.9, ups=1.51, wpb=31055, bsz=127.9, num_updates=37500, lr=0.001, gnorm=0.147, clip=0, train_wall=152, wall=14855 (progress_bar.py:262, log())
[2021-03-22 13:53:26] INFO >> epoch 020: 513 / 1973 loss=1.424, accuracy=0, mrr=0, ppl=2.68, wps=101626, ups=3.28, wpb=31029.3, bsz=128, num_updates=38000, lr=0.001, gnorm=0.176, clip=0, train_wall=151, wall=15008 (progress_bar.py:262, log())
[2021-03-22 13:56:00] INFO >> epoch 020: 1013 / 1973 loss=1.454, accuracy=0, mrr=0, ppl=2.74, wps=101419, ups=3.26, wpb=31141, bsz=128, num_updates=38500, lr=0.001, gnorm=0.144, clip=0, train_wall=152, wall=15161 (progress_bar.py:262, log())
[2021-03-22 13:58:34] INFO >> epoch 020: 1513 / 1973 loss=1.447, accuracy=0, mrr=0, ppl=2.73, wps=100560, ups=3.25, wpb=30919.5, bsz=128, num_updates=39000, lr=0.001, gnorm=0.142, clip=0, train_wall=153, wall=15315 (progress_bar.py:262, log())
[2021-03-22 14:00:55] INFO >> epoch 020 | loss 1.446 | accuracy 0 | mrr 0 | ppl 2.72 | wps 78249.5 | ups 2.52 | wpb 31027.9 | bsz 128 | num_updates 39460 | lr 0.001 | gnorm 0.153 | clip 0 | train_wall 600 | wall 15456 (progress_bar.py:269, print())
[2021-03-22 14:03:40] INFO >> epoch 020 | valid on 'test' subset | loss 2.072 | accuracy 0.598423 | mrr 0.674613 | ppl 4.21 | wps 186181 | wpb 61759.1 | bsz 255.8 | num_updates 39460 | best_mrr 0.674613 (progress_bar.py:269, print())
[2021-03-22 14:03:45] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 20 @ 39460 updates, score 0.674613) (writing took 5.136764 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 14:04:04] INFO >> epoch 021: 40 / 1973 loss=1.457, accuracy=0, mrr=0, ppl=2.75, wps=46898.3, ups=1.51, wpb=31006.5, bsz=127.9, num_updates=39500, lr=0.001, gnorm=0.149, clip=0, train_wall=152, wall=15646 (progress_bar.py:262, log())
[2021-03-22 14:06:38] INFO >> epoch 021: 540 / 1973 loss=1.4, accuracy=0, mrr=0, ppl=2.64, wps=101186, ups=3.26, wpb=31025.8, bsz=128, num_updates=40000, lr=0.001, gnorm=0.141, clip=0, train_wall=152, wall=15799 (progress_bar.py:262, log())
[2021-03-22 14:09:10] INFO >> epoch 021: 1040 / 1973 loss=1.423, accuracy=0, mrr=0, ppl=2.68, wps=102057, ups=3.29, wpb=31063.5, bsz=128, num_updates=40500, lr=0.001, gnorm=0.142, clip=0, train_wall=151, wall=15951 (progress_bar.py:262, log())
[2021-03-22 14:11:43] INFO >> epoch 021: 1540 / 1973 loss=1.435, accuracy=0, mrr=0, ppl=2.7, wps=101551, ups=3.27, wpb=31039, bsz=128, num_updates=41000, lr=0.001, gnorm=0.144, clip=0, train_wall=152, wall=16104 (progress_bar.py:262, log())
[2021-03-22 14:13:56] INFO >> epoch 021 | loss 1.423 | accuracy 0 | mrr 0 | ppl 2.68 | wps 78400.8 | ups 2.53 | wpb 31027.9 | bsz 128 | num_updates 41433 | lr 0.001 | gnorm 0.143 | clip 0 | train_wall 599 | wall 16237 (progress_bar.py:269, print())
[2021-03-22 14:16:41] INFO >> epoch 021 | valid on 'test' subset | loss 2.078 | accuracy 0.598087 | mrr 0.674371 | ppl 4.22 | wps 186741 | wpb 61759.1 | bsz 255.8 | num_updates 41433 | best_mrr 0.674613 (progress_bar.py:269, print())
[2021-03-22 14:16:44] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_last.pt (epoch 21 @ 41433 updates, score 0.674371) (writing took 2.961234 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 14:17:11] INFO >> epoch 022: 67 / 1973 loss=1.428, accuracy=0, mrr=0, ppl=2.69, wps=47213.7, ups=1.52, wpb=31020.5, bsz=127.9, num_updates=41500, lr=0.001, gnorm=0.143, clip=0, train_wall=152, wall=16432 (progress_bar.py:262, log())
[2021-03-22 14:19:45] INFO >> epoch 022: 567 / 1973 loss=1.385, accuracy=0, mrr=0, ppl=2.61, wps=100930, ups=3.26, wpb=31004.2, bsz=128, num_updates=42000, lr=0.001, gnorm=0.143, clip=0, train_wall=152, wall=16586 (progress_bar.py:262, log())
[2021-03-22 14:22:19] INFO >> epoch 022: 1067 / 1973 loss=1.409, accuracy=0, mrr=0, ppl=2.66, wps=100337, ups=3.24, wpb=31013.4, bsz=128, num_updates=42500, lr=0.001, gnorm=0.147, clip=0, train_wall=153, wall=16741 (progress_bar.py:262, log())
[2021-03-22 14:24:52] INFO >> epoch 022: 1567 / 1973 loss=1.422, accuracy=0, mrr=0, ppl=2.68, wps=101492, ups=3.26, wpb=31086.3, bsz=128, num_updates=43000, lr=0.001, gnorm=0.145, clip=0, train_wall=152, wall=16894 (progress_bar.py:262, log())
[2021-03-22 14:26:57] INFO >> epoch 022 | loss 1.409 | accuracy 0 | mrr 0 | ppl 2.66 | wps 78310 | ups 2.52 | wpb 31027.9 | bsz 128 | num_updates 43406 | lr 0.001 | gnorm 0.145 | clip 0 | train_wall 602 | wall 17019 (progress_bar.py:269, print())
[2021-03-22 14:29:43] INFO >> epoch 022 | valid on 'test' subset | loss 2.077 | accuracy 0.598166 | mrr 0.674426 | ppl 4.22 | wps 186338 | wpb 61759.1 | bsz 255.8 | num_updates 43406 | best_mrr 0.674613 (progress_bar.py:269, print())
[2021-03-22 14:29:46] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_last.pt (epoch 22 @ 43406 updates, score 0.674426) (writing took 3.152555 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 14:30:22] INFO >> epoch 023: 94 / 1973 loss=1.418, accuracy=0, mrr=0, ppl=2.67, wps=47055.2, ups=1.52, wpb=30975.6, bsz=127.9, num_updates=43500, lr=0.001, gnorm=0.145, clip=0, train_wall=152, wall=17223 (progress_bar.py:262, log())
[2021-03-22 14:32:54] INFO >> epoch 023: 594 / 1973 loss=1.378, accuracy=0, mrr=0, ppl=2.6, wps=101475, ups=3.27, wpb=31032.6, bsz=128, num_updates=44000, lr=0.001, gnorm=0.144, clip=0, train_wall=152, wall=17376 (progress_bar.py:262, log())
[2021-03-22 14:35:27] INFO >> epoch 023: 1094 / 1973 loss=1.395, accuracy=0, mrr=0, ppl=2.63, wps=101661, ups=3.27, wpb=31048.4, bsz=128, num_updates=44500, lr=0.001, gnorm=0.147, clip=0, train_wall=152, wall=17529 (progress_bar.py:262, log())
[2021-03-22 14:38:00] INFO >> epoch 023: 1594 / 1973 loss=1.407, accuracy=0, mrr=0, ppl=2.65, wps=101595, ups=3.27, wpb=31044.6, bsz=128, num_updates=45000, lr=0.001, gnorm=0.147, clip=0, train_wall=152, wall=17681 (progress_bar.py:262, log())
[2021-03-22 14:39:56] INFO >> epoch 023 | loss 1.396 | accuracy 0 | mrr 0 | ppl 2.63 | wps 78607.6 | ups 2.53 | wpb 31027.9 | bsz 128 | num_updates 45379 | lr 0.001 | gnorm 0.146 | clip 0 | train_wall 599 | wall 17798 (progress_bar.py:269, print())
[2021-03-22 14:42:41] INFO >> epoch 023 | valid on 'test' subset | loss 2.086 | accuracy 0.598494 | mrr 0.674641 | ppl 4.25 | wps 186551 | wpb 61759.1 | bsz 255.8 | num_updates 45379 | best_mrr 0.674641 (progress_bar.py:269, print())
[2021-03-22 14:42:46] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_best.pt (epoch 23 @ 45379 updates, score 0.674641) (writing took 4.546342 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 14:43:30] INFO >> epoch 024: 121 / 1973 loss=1.403, accuracy=0, mrr=0, ppl=2.64, wps=47035.1, ups=1.52, wpb=30997.7, bsz=127.9, num_updates=45500, lr=0.001, gnorm=0.144, clip=0, train_wall=152, wall=18011 (progress_bar.py:262, log())
[2021-03-22 14:46:03] INFO >> epoch 024: 621 / 1973 loss=1.368, accuracy=0, mrr=0, ppl=2.58, wps=101291, ups=3.26, wpb=31074.5, bsz=128, num_updates=46000, lr=0.001, gnorm=0.142, clip=0, train_wall=152, wall=18164 (progress_bar.py:262, log())
[2021-03-22 14:48:37] INFO >> epoch 024: 1121 / 1973 loss=1.379, accuracy=0, mrr=0, ppl=2.6, wps=100321, ups=3.24, wpb=30959.3, bsz=128, num_updates=46500, lr=0.001, gnorm=0.145, clip=0, train_wall=153, wall=18319 (progress_bar.py:262, log())
[2021-03-22 14:51:12] INFO >> epoch 024: 1621 / 1973 loss=1.396, accuracy=0, mrr=0, ppl=2.63, wps=100437, ups=3.24, wpb=31039.3, bsz=128, num_updates=47000, lr=0.001, gnorm=0.148, clip=0, train_wall=153, wall=18473 (progress_bar.py:262, log())
[2021-03-22 14:53:00] INFO >> epoch 024 | loss 1.384 | accuracy 0 | mrr 0 | ppl 2.61 | wps 78122.3 | ups 2.52 | wpb 31027.9 | bsz 128 | num_updates 47352 | lr 0.001 | gnorm 0.145 | clip 0 | train_wall 602 | wall 18581 (progress_bar.py:269, print())
[2021-03-22 14:55:45] INFO >> epoch 024 | valid on 'test' subset | loss 2.092 | accuracy 0.59809 | mrr 0.674152 | ppl 4.26 | wps 186193 | wpb 61759.1 | bsz 255.8 | num_updates 47352 | best_mrr 0.674641 (progress_bar.py:269, print())
[2021-03-22 14:55:49] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_last.pt (epoch 24 @ 47352 updates, score 0.674152) (writing took 3.281285 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 14:56:41] INFO >> epoch 025: 148 / 1973 loss=1.384, accuracy=0, mrr=0, ppl=2.61, wps=47096.8, ups=1.52, wpb=30993.1, bsz=127.9, num_updates=47500, lr=0.001, gnorm=0.145, clip=0, train_wall=152, wall=18802 (progress_bar.py:262, log())
[2021-03-22 14:59:14] INFO >> epoch 025: 648 / 1973 loss=1.354, accuracy=0, mrr=0, ppl=2.56, wps=101341, ups=3.27, wpb=31008.8, bsz=128, num_updates=48000, lr=0.001, gnorm=0.141, clip=0, train_wall=152, wall=18955 (progress_bar.py:262, log())
[2021-03-22 15:01:47] INFO >> epoch 025: 1148 / 1973 loss=1.374, accuracy=0, mrr=0, ppl=2.59, wps=101488, ups=3.26, wpb=31088, bsz=128, num_updates=48500, lr=0.001, gnorm=0.146, clip=0, train_wall=152, wall=19108 (progress_bar.py:262, log())
[2021-03-22 15:04:22] INFO >> epoch 025: 1648 / 1973 loss=1.385, accuracy=0, mrr=0, ppl=2.61, wps=99959.6, ups=3.23, wpb=30993.4, bsz=128, num_updates=49000, lr=0.001, gnorm=0.146, clip=0, train_wall=154, wall=19263 (progress_bar.py:262, log())
[2021-03-22 15:06:02] INFO >> epoch 025 | loss 1.372 | accuracy 0 | mrr 0 | ppl 2.59 | wps 78244.3 | ups 2.52 | wpb 31027.9 | bsz 128 | num_updates 49325 | lr 0.001 | gnorm 0.144 | clip 0 | train_wall 602 | wall 19364 (progress_bar.py:269, print())
[2021-03-22 15:08:47] INFO >> epoch 025 | valid on 'test' subset | loss 2.098 | accuracy 0.598019 | mrr 0.674096 | ppl 4.28 | wps 186304 | wpb 61759.1 | bsz 255.8 | num_updates 49325 | best_mrr 0.674641 (progress_bar.py:269, print())
[2021-03-22 15:08:50] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_last.pt (epoch 25 @ 49325 updates, score 0.674096) (writing took 2.893063 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 15:09:51] INFO >> epoch 026: 175 / 1973 loss=1.375, accuracy=0, mrr=0, ppl=2.59, wps=47233.4, ups=1.52, wpb=31059.6, bsz=127.9, num_updates=49500, lr=0.001, gnorm=0.143, clip=0, train_wall=152, wall=19592 (progress_bar.py:262, log())
[2021-03-22 15:12:25] INFO >> epoch 026: 675 / 1973 loss=1.343, accuracy=0, mrr=0, ppl=2.54, wps=100721, ups=3.25, wpb=31001.4, bsz=128, num_updates=50000, lr=0.001, gnorm=0.144, clip=0, train_wall=153, wall=19746 (progress_bar.py:262, log())
[2021-03-22 15:14:58] INFO >> epoch 026: 1175 / 1973 loss=1.362, accuracy=0, mrr=0, ppl=2.57, wps=101370, ups=3.26, wpb=31099.7, bsz=128, num_updates=50500, lr=0.001, gnorm=0.144, clip=0, train_wall=152, wall=19899 (progress_bar.py:262, log())
[2021-03-22 15:17:31] INFO >> epoch 026: 1675 / 1973 loss=1.376, accuracy=0, mrr=0, ppl=2.6, wps=101426, ups=3.27, wpb=31059.9, bsz=128, num_updates=51000, lr=0.001, gnorm=0.146, clip=0, train_wall=152, wall=20052 (progress_bar.py:262, log())
[2021-03-22 15:19:04] INFO >> epoch 026 | loss 1.361 | accuracy 0 | mrr 0 | ppl 2.57 | wps 78359.6 | ups 2.53 | wpb 31027.9 | bsz 128 | num_updates 51298 | lr 0.001 | gnorm 0.145 | clip 0 | train_wall 601 | wall 20145 (progress_bar.py:269, print())
[2021-03-22 15:21:49] INFO >> epoch 026 | valid on 'test' subset | loss 2.104 | accuracy 0.598392 | mrr 0.67424 | ppl 4.3 | wps 186633 | wpb 61759.1 | bsz 255.8 | num_updates 51298 | best_mrr 0.674641 (progress_bar.py:269, print())
[2021-03-22 15:21:52] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_last.pt (epoch 26 @ 51298 updates, score 0.67424) (writing took 2.957775 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 15:23:01] INFO >> epoch 027: 202 / 1973 loss=1.356, accuracy=0, mrr=0, ppl=2.56, wps=46991.7, ups=1.52, wpb=30961.6, bsz=127.9, num_updates=51500, lr=0.001, gnorm=0.144, clip=0, train_wall=153, wall=20382 (progress_bar.py:262, log())
[2021-03-22 15:25:34] INFO >> epoch 027: 702 / 1973 loss=1.333, accuracy=0, mrr=0, ppl=2.52, wps=101374, ups=3.27, wpb=31015.2, bsz=128, num_updates=52000, lr=0.001, gnorm=0.142, clip=0, train_wall=152, wall=20535 (progress_bar.py:262, log())
[2021-03-22 15:28:07] INFO >> epoch 027: 1202 / 1973 loss=1.357, accuracy=0, mrr=0, ppl=2.56, wps=100750, ups=3.26, wpb=30947.7, bsz=128, num_updates=52500, lr=0.001, gnorm=0.143, clip=0, train_wall=152, wall=20688 (progress_bar.py:262, log())
[2021-03-22 15:30:40] INFO >> epoch 027: 1702 / 1973 loss=1.364, accuracy=0, mrr=0, ppl=2.57, wps=101637, ups=3.27, wpb=31034.9, bsz=128, num_updates=53000, lr=0.001, gnorm=0.145, clip=0, train_wall=152, wall=20841 (progress_bar.py:262, log())
[2021-03-22 15:32:03] INFO >> epoch 027 | loss 1.351 | accuracy 0 | mrr 0 | ppl 2.55 | wps 78544 | ups 2.53 | wpb 31027.9 | bsz 128 | num_updates 53271 | lr 0.001 | gnorm 0.143 | clip 0 | train_wall 600 | wall 20924 (progress_bar.py:269, print())
[2021-03-22 15:34:48] INFO >> epoch 027 | valid on 'test' subset | loss 2.111 | accuracy 0.597692 | mrr 0.673783 | ppl 4.32 | wps 187089 | wpb 61759.1 | bsz 255.8 | num_updates 53271 | best_mrr 0.674641 (progress_bar.py:269, print())
[2021-03-22 15:34:51] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_last.pt (epoch 27 @ 53271 updates, score 0.673783) (writing took 3.277195 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 15:36:08] INFO >> epoch 028: 229 / 1973 loss=1.344, accuracy=0, mrr=0, ppl=2.54, wps=47483.4, ups=1.52, wpb=31150.9, bsz=127.9, num_updates=53500, lr=0.001, gnorm=0.142, clip=0, train_wall=152, wall=21169 (progress_bar.py:262, log())
[2021-03-22 15:38:41] INFO >> epoch 028: 729 / 1973 loss=1.326, accuracy=0, mrr=0, ppl=2.51, wps=101310, ups=3.27, wpb=30988.8, bsz=128, num_updates=54000, lr=0.001, gnorm=0.143, clip=0, train_wall=152, wall=21322 (progress_bar.py:262, log())
[2021-03-22 15:41:14] INFO >> epoch 028: 1229 / 1973 loss=1.344, accuracy=0, mrr=0, ppl=2.54, wps=101386, ups=3.27, wpb=31035.6, bsz=128, num_updates=54500, lr=0.001, gnorm=0.147, clip=0, train_wall=152, wall=21475 (progress_bar.py:262, log())
[2021-03-22 15:43:47] INFO >> epoch 028: 1729 / 1973 loss=1.36, accuracy=0, mrr=0, ppl=2.57, wps=101176, ups=3.26, wpb=31058.5, bsz=128, num_updates=55000, lr=0.001, gnorm=0.144, clip=0, train_wall=152, wall=21629 (progress_bar.py:262, log())
[2021-03-22 15:45:02] INFO >> epoch 028 | loss 1.342 | accuracy 0 | mrr 0 | ppl 2.53 | wps 78556.1 | ups 2.53 | wpb 31027.9 | bsz 128 | num_updates 55244 | lr 0.001 | gnorm 0.145 | clip 0 | train_wall 599 | wall 21704 (progress_bar.py:269, print())
[2021-03-22 15:47:48] INFO >> epoch 028 | valid on 'test' subset | loss 2.117 | accuracy 0.597801 | mrr 0.673804 | ppl 4.34 | wps 186137 | wpb 61759.1 | bsz 255.8 | num_updates 55244 | best_mrr 0.674641 (progress_bar.py:269, print())
[2021-03-22 15:47:51] INFO >> saved checkpoint /home/wanyao/.ncc/raw_py150/completion/data-mmap/seqrnn/checkpoints/checkpoint_last.pt (epoch 28 @ 55244 updates, score 0.673804) (writing took 2.879432 seconds) (checkpoint_utils.py:81, save_checkpoint())
[2021-03-22 15:47:51] INFO >> early stop since valid performance hasn't improved for last 5 runs (train.py:185, should_stop_early())
[2021-03-22 15:47:51] INFO >> early stop since valid performance hasn't improved for last 5 runs (train.py:271, single_main())
[2021-03-22 15:47:51] INFO >> done training in 21870.5 seconds (train.py:282, single_main())