Skip to main content
eleutherai
Projects
pythia
Run groups
v2-12B-deduped_o4mu97sj
Logs
Group
Workspace
Runs
Logs
Files
Artifacts

Atmallen8's group workspace

Timestamps visible
2023-02-23 14:10:40
Using ./extensions/ as PyTorch extensions root...
2023-02-23 14:10:40
No modifications detected for re-loaded extension module utils, skipping build step...
2023-02-23 14:10:40
Loading extension module utils...
2023-02-23 14:10:44
Traceback (most recent call last):
2023-02-23 14:10:44
  File "/fsx/gpt-neox/train.py", line 27, in <module>
2023-02-23 14:10:44
    pretrain(neox_args=neox_args)
2023-02-23 14:10:44
  File "/fsx/gpt-neox/megatron/training.py", line 96, in pretrain
2023-02-23 14:10:44
    ) = build_train_valid_test_data_iterators(neox_args=neox_args)
2023-02-23 14:10:44
  File "/fsx/gpt-neox/megatron/data/data_utils.py", line 329, in build_train_valid_test_data_iterators
2023-02-23 14:10:44
    train_datasets, valid_datasets, test_datasets = build_weighted_datasets(
2023-02-23 14:10:44
  File "/fsx/gpt-neox/megatron/data/data_utils.py", line 210, in build_weighted_datasets
2023-02-23 14:10:44
    build_the_dataset(
2023-02-23 14:10:44
  File "/fsx/gpt-neox/megatron/data/data_utils.py", line 68, in build_the_dataset
2023-02-23 14:10:44
    total_num_of_documents = indexed_dataset.sizes.shape[0]
2023-02-23 14:10:44
AttributeError: 'NoneType' object has no attribute 'sizes'
2023-02-23 14:10:44
[2023-02-23 14:10:43,734] [WARNING] [engine.py:1519:load_checkpoint] Unable to find latest file at /fsx/hailey/pythia/new_ckpts/pythia-v2-12b-deduped/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
2023-02-23 14:10:44
Dataset does not exist: /fsx/pile_deduped/pile_0.87_deduped_text_document
2023-02-23 14:10:44
Path should be a basename that both .idx and .bin can be appended to get full filenames.