eleutherai

Preetham-gali's group workspace

Group: cawcJ6TCRPjouBa97Rtb3K_3jx6l7yy

2760

1-1

of 1

Timestamps visible

2021-08-30 17:37:27

[2021-08-30 17:37:25,894] [INFO] [stage1.py:695:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 64.0, reducing to 32.0

2021-08-30 17:37:33

[2021-08-30 17:37:30,968] [INFO] [stage1.py:695:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 32.0, reducing to 16.0

2021-08-30 17:37:37

[2021-08-30 17:37:36,033] [INFO] [stage1.py:695:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 16.0, reducing to 8.0

2021-08-30 17:37:43

[2021-08-30 17:37:41,095] [INFO] [stage1.py:695:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 8.0, reducing to 4.0

2021-08-30 17:37:47

[2021-08-30 17:37:46,153] [INFO] [stage1.py:695:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 4.0, reducing to 2.0

2021-08-30 17:37:47

[2021-08-30 17:37:46,153] [INFO] [logging.py:60:log_dist] [Rank 0] step=3370, skipped=34, lr=[0.0002999917226836827, 0.0002999917226836827], mom=[[0.9, 0.999], [0.9, 0.999]]

2021-08-30 17:37:47

steps: 3370 loss: nan iter time (s): 5.099 samples/sec: 37.654

2021-08-30 17:37:47

%comms: 4.998656894437888

2021-08-30 17:37:47

%optimizer_step 0.5472861763086839

2021-08-30 17:37:47

%forward: 39.56839670975842

2021-08-30 17:37:47

%backward: 45.39121730999124

2021-08-30 17:37:47

2021-08-30 17:37:53

[2021-08-30 17:37:51,213] [INFO] [stage1.py:695:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 2.0, reducing to 1.0