Skip to main content
eleutherai
Projects
distilling
Run groups
PWV7zu6gsBVbcrVgbMzpqN_1u29y8gl
Logs
Group
Workspace
Runs
Logs
Files
Artifacts

Preetham-gali's group workspace

Timestamps visible
2021-08-16 18:53:03
 %backward: 53.12344190943088
2021-08-16 18:53:03
[2021-08-16 18:53:03,207] [INFO] [logging.py:60:log_dist] [Rank 0] rank=0 time (ms) | train_batch: 0.00 | batch_input: 779.79 | forward: 45627.92 | backward_microstep: 77982.06 | backward: 77972.48 | backward_inner_microstep: 77942.92 | backward_inner: 77932.59 | backward_allreduce_microstep: 14.80 | backward_allreduce: 5.19 | reduce_tied_grads: 0.58 | comms: 278.83 | reduce_grads: 233.58 | step: 426.93 | _step_clipping: 0.14 | _step_step: 424.33 | _step_zero_grad: 0.93 | _step_check_overflow: 0.78
2021-08-16 18:53:03
 samples/sec: 36.791 | iteration     5300/  250000 | elapsed time per iteration (ms): 14677.4 | learning rate: 2.999E-04 | approx flops per GPU: 131.4TFLOPS | loss: 4.518149E+00 | lm_loss: 3.599088E+00 | kld_loss: 9.190623E-01 | mse_loss: 0.000000E+00 | loss scale: 16384.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
2021-08-16 18:53:03
time (ms)
2021-08-16 18:55:30
[2021-08-16 18:55:29,988] [INFO] [logging.py:60:log_dist] [Rank 0] step=5310, skipped=25, lr=[0.00029990814832879986, 0.00029990814832879986], mom=[[0.9, 0.999], [0.9, 0.999]]
2021-08-16 18:55:30
steps: 5310 loss: 4.5393 iter time (s): 14.676 samples/sec: 36.796
2021-08-16 18:55:30
%comms: 0.19171515089996552
2021-08-16 18:55:30
 %optimizer_step 0.29203667948124606
2021-08-16 18:55:30
 %forward: 31.074316689401027
2021-08-16 18:55:30
 %backward: 53.12235281507428
2021-08-16 18:55:30
[2021-08-16 18:55:29,990] [INFO] [logging.py:60:log_dist] [Rank 0] rank=0 time (ms) | train_batch: 0.00 | batch_input: 790.16 | forward: 45603.62 | backward_microstep: 77970.12 | backward: 77960.57 | backward_inner_microstep: 77931.70 | backward_inner: 77921.57 | backward_allreduce_microstep: 14.48 | backward_allreduce: 5.07 | reduce_tied_grads: 0.69 | comms: 281.35 | reduce_grads: 235.74 | step: 428.58 | _step_clipping: 0.14 | _step_step: 426.14 | _step_zero_grad: 0.83 | _step_check_overflow: 0.76