eleutherai

Chilli's group workspace

Group: nyBs7KmXWhqwBzVfPxqGwn_f8cuuu1q

646

1-8

of 8

Timestamps visible

2023-08-03 04:57:08

[2023-08-03 04:57:07,663] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py

2023-08-03 04:57:08

[2023-08-03 04:57:07,757] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38005/zero_pp_rank_16_mp_rank_00_optim_states.pt

2023-08-03 04:57:23

[2023-08-03 04:57:23,090] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py

2023-08-03 04:57:23

[2023-08-03 04:57:23,139] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38006/zero_pp_rank_16_mp_rank_00_optim_states.pt

2023-08-03 04:57:39

[2023-08-03 04:57:38,720] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py

2023-08-03 04:57:39

[2023-08-03 04:57:38,750] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38007/zero_pp_rank_16_mp_rank_00_optim_states.pt

2023-08-03 04:58:01

[2023-08-03 04:58:00,504] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py

2023-08-03 04:58:01

[2023-08-03 04:58:00,512] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38008/zero_pp_rank_16_mp_rank_00_optim_states.pt

2023-08-03 04:58:21

[2023-08-03 04:58:21,624] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py

2023-08-03 04:58:21

[2023-08-03 04:58:21,628] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38009/zero_pp_rank_16_mp_rank_00_optim_states.pt

2023-08-03 04:58:46

[2023-08-03 04:58:44,768] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py

2023-08-03 04:58:46

[2023-08-03 04:58:44,847] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38010/zero_pp_rank_16_mp_rank_00_optim_states.pt