Skip to main content

Chilli's group workspace

Timestamps visible
2023-08-03 07:50:26
No modifications detected for re-loaded extension module utils, skipping build step...
2023-08-03 07:50:26
Loading extension module utils...
2023-08-03 07:50:26
Time to load utils op: 0.0017795562744140625 seconds
2023-08-03 07:50:28
[2023-08-03 07:50:27,207] [INFO] [engine.py:1551:_load_checkpoint] rank: 24 loading checkpoint: /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38002/mp_rank_00_model_states.pt
2023-08-03 07:52:19
successfully loaded 64 ZeRO state_dicts for rank 24
2023-08-03 07:52:50
loading 64 zero partition checkpoints for rank 24
2023-08-03 07:53:08
WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)
2023-08-03 07:53:10
WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)
2023-08-03 07:53:12
WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)
2023-08-03 07:53:14
> RANK 24 elapsed time for building blendable dataset indices: 0.59 (sec)
2023-08-03 07:53:14
> RANK 24 elapsed time for building blendable dataset indices: 1.04 (sec)
2023-08-03 07:53:17
> RANK 24 elapsed time for building blendable dataset indices: 1.10 (sec)
2023-08-03 07:53:47
[2023-08-03 07:53:46,400] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py
2023-08-03 07:53:47
[2023-08-03 07:53:46,408] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38003/zero_pp_rank_24_mp_rank_00_optim_states.pt
2023-08-03 07:54:01
[2023-08-03 07:54:01,106] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py
2023-08-03 07:54:01
[2023-08-03 07:54:01,230] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38004/zero_pp_rank_24_mp_rank_00_optim_states.pt