Skip to main content
eleutherai
Projects
neox
Run groups
JrUmQ3ETsY5zQbun8VYKrd_1oddvd4l
Logs
Overview
Workspace
Runs
Logs
Files
Artifacts

Chilli's group workspace

Timestamps visible
2023-08-03 08:26:01
Using ./extensions/ as PyTorch extensions root...
2023-08-03 08:26:01
No modifications detected for re-loaded extension module utils, skipping build step...
2023-08-03 08:26:01
Loading extension module utils...
2023-08-03 08:26:03
[2023-08-03 08:26:01,922] [INFO] [engine.py:1551:_load_checkpoint] rank: 8 loading checkpoint: /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38002/mp_rank_00_model_states.pt
2023-08-03 08:27:53
successfully loaded 64 ZeRO state_dicts for rank 8
2023-08-03 08:28:24
loading 64 zero partition checkpoints for rank 8
2023-08-03 08:28:44
WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)
2023-08-03 08:28:46
WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)
2023-08-03 08:28:48
WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)
2023-08-03 08:28:50
> RANK 8 elapsed time for building blendable dataset indices: 0.59 (sec)
2023-08-03 08:28:50
> RANK 8 elapsed time for building blendable dataset indices: 1.17 (sec)
2023-08-03 08:28:53
> RANK 8 elapsed time for building blendable dataset indices: 1.26 (sec)
2023-08-03 08:29:25
[2023-08-03 08:29:24,986] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py
2023-08-03 08:29:25
[2023-08-03 08:29:25,015] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38003/zero_pp_rank_8_mp_rank_00_optim_states.pt
2023-08-03 08:29:39
[2023-08-03 08:29:39,236] [INFO] [engine.py:1805:_copy_recovery_script] creating recovery script /fsx/lintangsutawika/checkpoints/temp_neox_models/zero_to_fp32.py
2023-08-03 08:29:39
[2023-08-03 08:29:39,291] [INFO] [engine.py:1818:_save_zero_checkpoint] zero checkpoint saved /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38004/zero_pp_rank_8_mp_rank_00_optim_states.pt