eleutherai

Igoro's group workspace

Group: math_25yr7dbz

379

1-8

of 8

Timestamps visible

2022-02-18 15:57:16

[2022-02-18 15:57:15,286] [INFO] [stage1.py:382:get_data_parallel_sub_partitions] ****

2022-02-18 15:57:16

[2022-02-18 15:57:15,325] [INFO] [stage1.py:375:get_data_parallel_sub_partitions] **** partition info:

2022-02-18 15:57:16

[2022-02-18 15:57:15,325] [INFO] [stage1.py:376:get_data_parallel_sub_partitions] total_num_elements=700416

2022-02-18 15:57:16

[2022-02-18 15:57:15,331] [INFO] [stage1.py:377:get_data_parallel_sub_partitions] world_size=8

2022-02-18 15:57:16

[2022-02-18 15:57:15,337] [INFO] [stage1.py:378:get_data_parallel_sub_partitions] max_elements_per_comm=700416

2022-02-18 15:57:16

[2022-02-18 15:57:15,337] [INFO] [stage1.py:379:get_data_parallel_sub_partitions] sub_partition_size=87552

2022-02-18 15:57:16

[2022-02-18 15:57:15,337] [INFO] [stage1.py:380:get_data_parallel_sub_partitions] num_sub_partitions=8

2022-02-18 15:57:16

[2022-02-18 15:57:15,337] [INFO] [stage1.py:381:get_data_parallel_sub_partitions] num_comm_intervals=1

2022-02-18 15:57:16

[2022-02-18 15:57:15,337] [INFO] [stage1.py:382:get_data_parallel_sub_partitions] ****

2022-02-18 15:57:18

Using /home/mchorse/.cache/torch_extensions as PyTorch extensions root...

2022-02-18 15:57:18

No modifications detected for re-loaded extension module utils, skipping build step...

2022-02-18 15:57:18

Loading extension module utils...

2022-02-18 15:57:18

Time to load utils op: 0.002660989761352539 seconds

2022-02-18 15:57:20

[2022-02-18 15:57:19,887] [INFO] [engine.py:141:__init__] RANK=16 STAGE=1 LAYERS=12 [13, 25) STAGE_PARAMS=2718609408 (2718.609M) TOTAL_PARAMS=20556201984 (20556.202M) UNIQUE_PARAMS=20556201984 (20556.202M)

2022-02-18 15:57:26

> number of parameters on model parallel rank 0: 2718609408

2022-02-18 15:57:34

[2022-02-18 15:57:32,343] [WARNING] [engine.py:1519:load_checkpoint] Unable to find latest file at /mnt/ssd-1/20B_math/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.

2022-02-18 15:57:34

Unable to load checkpoint.