Skip to main content
eleutherai
Projects
neox
Run groups
V2VUtaNzTaJkQwhi66oHfL_22z49l2b
Logs
Overview
Workspace
Runs
Logs
Files
Artifacts

Chilli's group workspace

Timestamps visible
2023-08-03 09:02:39
Using ./extensions/ as PyTorch extensions root...
2023-08-03 09:02:39
No modifications detected for re-loaded extension module utils, skipping build step...
2023-08-03 09:02:39
Loading extension module utils...
2023-08-03 09:02:41
[2023-08-03 09:02:41,330] [INFO] [engine.py:1551:_load_checkpoint] rank: 16 loading checkpoint: /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38000/mp_rank_00_model_states.pt
2023-08-03 09:04:41
successfully loaded 64 ZeRO state_dicts for rank 16
2023-08-03 09:05:10
loading 64 zero partition checkpoints for rank 16
2023-08-03 09:05:26
WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)
2023-08-03 09:05:28
WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)
2023-08-03 09:05:32
WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)
2023-08-03 09:05:32
> RANK 16 elapsed time for building blendable dataset indices: 0.65 (sec)
2023-08-03 09:05:32
> RANK 16 elapsed time for building blendable dataset indices: 1.10 (sec)
2023-08-03 09:05:34
> RANK 16 elapsed time for building blendable dataset indices: 1.18 (sec)
2023-08-03 09:06:06
[2023-08-03 09:06:05,712] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py
2023-08-03 09:06:06
[2023-08-03 09:06:05,779] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38001/zero_pp_rank_16_mp_rank_00_optim_states.pt
2023-08-03 09:06:21
[2023-08-03 09:06:20,897] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py
2023-08-03 09:06:21
[2023-08-03 09:06:20,954] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38002/zero_pp_rank_16_mp_rank_00_optim_states.pt