eleutherai

Preetham-gali's group workspace

Group: VNEXM4EhL7ZZTKv2RCSFmm_yqc7d4l9

491

1-1

of 1

Timestamps visible

2021-09-02 16:53:07

2021-09-02 16:53:07

time (ms)

2021-09-02 16:53:13

[2021-09-02 16:53:12,273] [INFO] [stage1.py:695:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 16.0, reducing to 8.0

2021-09-02 16:53:19

[2021-09-02 16:53:17,961] [INFO] [stage1.py:695:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 8.0, reducing to 4.0

2021-09-02 16:53:29

[2021-09-02 16:53:29,411] [INFO] [stage1.py:695:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 4.0, reducing to 2.0

2021-09-02 16:53:48

[2021-09-02 16:53:46,609] [INFO] [stage1.py:695:step] [deepspeed] fp16 dynamic loss scale overflow! Skipping step. Attempted loss scale: 2.0, reducing to 1.0

2021-09-02 16:54:04

[2021-09-02 16:54:03,858] [INFO] [logging.py:60:log_dist] [Rank 0] step=210, skipped=33, lr=[2.1239999999999997e-05, 2.1239999999999997e-05], mom=[[0.9, 0.999], [0.9, 0.999]]

2021-09-02 16:54:04

steps: 210 loss: 9.8110 iter time (s): 5.726 samples/sec: 90.817

2021-09-02 16:54:04

%comms: 2.676491266278921

2021-09-02 16:54:04

%optimizer_step 0.7711798505281412

2021-09-02 16:54:04

%forward: 44.04209475061005

2021-09-02 16:54:04

%backward: 39.007011530686725

2021-09-02 16:54:04