Skip to main content
eleutherai
Projects
neox
Run groups
SWDKQ6x8Rs5E2xzRZ9NxiV
Logs
Overview
Workspace
Runs
Logs
Files
Artifacts

Chilli's group workspace

Timestamps visible
2021-03-26 16:05:49
[2021-03-26 16:05:49,060] [INFO] [unfused_optimizer.py:244:_update_scale] Reducing dynamic loss scale from 65536.0 to 32768.0
2021-03-26 16:05:49
[2021-03-26 16:05:49,060] [INFO] [unfused_optimizer.py:168:step] [deepspeed] OVERFLOW! Skipping step. Attempted loss scale: 65536.0, reducing to 32768.0
2021-03-26 16:05:51
[2021-03-26 16:05:51,354] [INFO] [unfused_optimizer.py:243:_update_scale] Grad overflow on iteration: 17
2021-03-26 16:05:51
[2021-03-26 16:05:51,355] [INFO] [unfused_optimizer.py:244:_update_scale] Reducing dynamic loss scale from 32768.0 to 16384.0
2021-03-26 16:05:51
[2021-03-26 16:05:51,355] [INFO] [unfused_optimizer.py:168:step] [deepspeed] OVERFLOW! Skipping step. Attempted loss scale: 32768.0, reducing to 16384.0
2021-03-26 16:06:13
[2021-03-26 16:06:13,351] [INFO] [unfused_optimizer.py:243:_update_scale] Grad overflow on iteration: 26
2021-03-26 16:06:13
[2021-03-26 16:06:13,351] [INFO] [unfused_optimizer.py:244:_update_scale] Reducing dynamic loss scale from 16384.0 to 8192.0
2021-03-26 16:06:13
[2021-03-26 16:06:13,351] [INFO] [unfused_optimizer.py:168:step] [deepspeed] OVERFLOW! Skipping step. Attempted loss scale: 16384.0, reducing to 8192.0
2021-03-26 16:49:24
[2021-03-26 16:49:24,183] [INFO] [unfused_optimizer.py:253:_update_scale] No Grad overflow for 1000 iterations
2021-03-26 16:49:24
[2021-03-26 16:49:24,184] [INFO] [unfused_optimizer.py:255:_update_scale] Increasing dynamic loss scale from 8192.0 to 16384.0
2021-03-26 16:50:20
[2021-03-26 16:50:20,375] [INFO] [unfused_optimizer.py:243:_update_scale] Grad overflow on iteration: 1049
2021-03-26 16:50:20
[2021-03-26 16:50:20,375] [INFO] [unfused_optimizer.py:244:_update_scale] Reducing dynamic loss scale from 16384.0 to 8192.0
2021-03-26 16:50:20
[2021-03-26 16:50:20,375] [INFO] [unfused_optimizer.py:168:step] [deepspeed] OVERFLOW! Skipping step. Attempted loss scale: 16384.0, reducing to 8192.0