eleutherai

Chilli's group workspace

Group: AguqEyBWoYohy56s5MRsgV_2iqk7d37

502

1-8

of 8

Timestamps visible

2023-08-03 09:16:32

/fsx/lintangsutawika/miniconda3/envs/pythia/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:429: UserWarning: torch.distributed.distributed_c10d._get_global_rank is deprecated please use torch.distributed.distributed_c10d.get_global_rank instead

2023-08-03 09:16:32

warnings.warn(

2023-08-03 09:16:34

Using ./extensions/ as PyTorch extensions root...

2023-08-03 09:16:34

Loading extension module utils...

2023-08-03 09:16:34

Time to load utils op: 0.6049346923828125 seconds

2023-08-03 09:16:34

[2023-08-03 09:16:34,758] [INFO] [stage1.py:160:__init__] ZeRO Elastic Checkpoint = True

2023-08-03 09:16:36

Using ./extensions/ as PyTorch extensions root...

2023-08-03 09:16:36

No modifications detected for re-loaded extension module utils, skipping build step...

2023-08-03 09:16:36

Loading extension module utils...

2023-08-03 09:16:36

Time to load utils op: 0.0024442672729492188 seconds

2023-08-03 09:16:38

[2023-08-03 09:16:37,768] [INFO] [engine.py:1551:_load_checkpoint] rank: 24 loading checkpoint: /fsx/lintangsutawika/checkpoints/temp_neox_models/global_step38004/mp_rank_00_model_states.pt

2023-08-03 09:18:14

successfully loaded 64 ZeRO state_dicts for rank 24

2023-08-03 09:18:50

loading 64 zero partition checkpoints for rank 24

2023-08-03 09:19:08

WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)

2023-08-03 09:19:10

WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)

2023-08-03 09:19:14

WARNING: shuffle index length (162165685) is not equal to sample index length (162165686)

2023-08-03 09:19:14

> RANK 24 elapsed time for building blendable dataset indices: 0.60 (sec)

2023-08-03 09:19:14

> RANK 24 elapsed time for building blendable dataset indices: 1.00 (sec)

2023-08-03 09:19:16

> RANK 24 elapsed time for building blendable dataset indices: 1.07 (sec)