Skip to main content

Chilli's group workspace

Timestamps visible
2023-08-03 08:07:44
Using ./extensions/ as PyTorch extensions root...
2023-08-03 08:07:44
No modifications detected for re-loaded extension module utils, skipping build step...
2023-08-03 08:07:44
Loading extension module utils...
2023-08-03 08:07:46
[2023-08-03 08:07:45,565] [INFO] [engine.py:1551:_load_checkpoint] rank: 40 loading checkpoint: /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38002/mp_rank_00_model_states.pt
2023-08-03 08:09:24
successfully loaded 64 ZeRO state_dicts for rank 40
2023-08-03 08:10:10
loading 64 zero partition checkpoints for rank 40
2023-08-03 08:10:20
WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)
2023-08-03 08:10:22
WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)
2023-08-03 08:10:26
WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)
2023-08-03 08:10:26
> RANK 40 elapsed time for building blendable dataset indices: 0.61 (sec)
2023-08-03 08:10:26
> RANK 40 elapsed time for building blendable dataset indices: 1.06 (sec)
2023-08-03 08:10:29
> RANK 40 elapsed time for building blendable dataset indices: 1.13 (sec)
2023-08-03 08:10:59
[2023-08-03 08:10:59,124] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py
2023-08-03 08:10:59
[2023-08-03 08:10:59,140] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38003/zero_pp_rank_40_mp_rank_00_optim_states.pt
2023-08-03 08:11:15
[2023-08-03 08:11:13,569] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py
2023-08-03 08:11:15
[2023-08-03 08:11:13,574] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38004/zero_pp_rank_40_mp_rank_00_optim_states.pt