eleutherai

Chilli's group workspace

Group: E9vFSoZCFToC5bH2FMgoB8_1tx52pbh

586

1-8

of 8

Timestamps visible

2023-08-03 05:13:44

> RANK 24 elapsed time for building blendable dataset indices: 0.98 (sec)

2023-08-03 05:13:46

> RANK 24 elapsed time for building blendable dataset indices: 1.05 (sec)

2023-08-03 05:14:19

[2023-08-03 05:14:17,421] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py

2023-08-03 05:14:19

[2023-08-03 05:14:17,431] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38001/zero_pp_rank_24_mp_rank_00_optim_states.pt

2023-08-03 05:14:33

[2023-08-03 05:14:32,541] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py

2023-08-03 05:14:33

[2023-08-03 05:14:32,647] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38002/zero_pp_rank_24_mp_rank_00_optim_states.pt

2023-08-03 05:14:47

[2023-08-03 05:14:47,181] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py

2023-08-03 05:14:47

[2023-08-03 05:14:47,192] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38003/zero_pp_rank_24_mp_rank_00_optim_states.pt

2023-08-03 05:15:04

[2023-08-03 05:15:02,121] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py

2023-08-03 05:15:04

[2023-08-03 05:15:02,196] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38004/zero_pp_rank_24_mp_rank_00_optim_states.pt

2023-08-03 05:15:18

[2023-08-03 05:15:16,343] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py

2023-08-03 05:15:18

[2023-08-03 05:15:16,440] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38005/zero_pp_rank_24_mp_rank_00_optim_states.pt