Skip to main content
eleutherai
Projects
neox
Run groups
bCkuQyUw6qN2rznmPX7FX7_3ldwc2e7
Logs
Overview
Workspace
Runs
Logs
Files
Artifacts

Chilli's group workspace

Timestamps visible
2023-08-03 08:03:22
Using ./extensions/ as PyTorch extensions root...
2023-08-03 08:03:22
No modifications detected for re-loaded extension module utils, skipping build step...
2023-08-03 08:03:22
Loading extension module utils...
2023-08-03 08:03:24
[2023-08-03 08:03:23,179] [INFO] [engine.py:1551:_load_checkpoint] rank: 40 loading checkpoint: /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38000/mp_rank_00_model_states.pt
2023-08-03 08:04:26
successfully loaded 64 ZeRO state_dicts for rank 40
2023-08-03 08:04:56
loading 64 zero partition checkpoints for rank 40
2023-08-03 08:05:48
WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)
2023-08-03 08:05:50
WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)
2023-08-03 08:05:54
WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)
2023-08-03 08:05:54
> RANK 40 elapsed time for building blendable dataset indices: 1.08 (sec)
2023-08-03 08:05:56
> RANK 40 elapsed time for building blendable dataset indices: 1.17 (sec)
2023-08-03 08:05:56
> RANK 40 elapsed time for building blendable dataset indices: 1.18 (sec)
2023-08-03 08:06:27
[2023-08-03 08:06:26,466] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py
2023-08-03 08:06:27
[2023-08-03 08:06:26,497] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38001/zero_pp_rank_40_mp_rank_00_optim_states.pt
2023-08-03 08:06:43
[2023-08-03 08:06:41,346] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py
2023-08-03 08:06:43
[2023-08-03 08:06:41,411] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38002/zero_pp_rank_40_mp_rank_00_optim_states.pt