eleutherai

Chilli's group workspace

Group: SWZ3VbtNxvzsfqPZQyeiyd_1xllln5d

214

238

218

9073

222

243

224

237

1-8

of 8

Timestamps visible

2022-03-09 09:48:08

[2022-03-09 09:48:08,261] [INFO] [stage1.py:697:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 32768.0, reducing to 16384.0

2022-03-09 09:48:37

[2022-03-09 09:48:37,006] [INFO] [stage1.py:697:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 16384.0, reducing to 8192.0

2022-03-09 09:49:06

[2022-03-09 09:49:05,492] [INFO] [stage1.py:697:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 8192.0, reducing to 4096.0

2022-03-09 19:00:01

[2022-03-09 19:00:01,076] [INFO] [stage1.py:697:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 8192.0, reducing to 8192.0

2022-03-09 19:15:41

[2022-03-09 19:15:41,280] [INFO] [stage1.py:697:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 8192.0, reducing to 4096.0

2022-03-10 00:12:33

[2022-03-10 00:12:32,672] [INFO] [stage1.py:697:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 4096.0, reducing to 2048.0

2022-03-10 01:07:50

[2022-03-10 01:07:49,808] [INFO] [stage1.py:697:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 2048.0, reducing to 1024.0

2022-03-10 09:33:31

wandb: Network error resolved after 0:00:57.161447, resuming normal operation.

2022-03-10 10:55:16

[2022-03-10 10:55:16,108] [INFO] [stage1.py:697:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 2048.0, reducing to 2048.0

2022-03-10 11:07:40

[2022-03-10 11:07:39,581] [INFO] [stage1.py:697:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 2048.0, reducing to 1024.0

2022-03-10 11:13:00

[2022-03-10 11:13:00,042] [INFO] [stage1.py:697:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 1024.0, reducing to 512.0

2022-03-10 11:43:42

[2022-03-10 11:43:41,900] [INFO] [stage1.py:697:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 512.0, reducing to 256.0