kastan

Kastan's group workspace

Group: Jul-27__17:01

1-1

of 1

Tags

my_first_tag

Author

kastan

State

Crashed

Start time

July 27th, 2022 10:01:41 PM

Runtime

4m 37s

Tracked hours

Run path

kastan/col_ai/z9ca8rms

Linux-4.18.0-305.49.1.el8_4.x86_64-x86_64-with-glibc2.28

Python version

3.9.12

Command

/u/kastanday/LLM-Distributed-Quantization/benchmarks/gpt/train.py --config /u/kastanday/LLM-Distributed-Quantization/benchmarks/gpt/configs/gpt2_8b_2p5d_256.py --host gpub051 --port 29500 --world_size 32 --rank 17

System Hardware

CPU count	64
GPU count	4
GPU type	NVIDIA A40

W&B CLI Version

0.12.21

Group

Jul-27__17:01

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 18 keys
- BATCH_SIZE:
  1,280
- clip_grad_norm:
  1
- data_dir:
  "/u/kastanday/colossal_ai/raw_json_backup/train_data_FINAL.json"
- ▶
  fp16:{} 1 key
  - mode:
    "AMP_TYPE.NAIVE"
- gpt2_8B:
  "titans.model.gpt.gpt.gpt2_8B"
- gpt2_xl:
  "titans.model.gpt.gpt.gpt2_xl"
- gradient_accumulation:
  1
- LEARNING_RATE:
  0.00015
- ▶
  model:{} 5 keys
- NUM_EPOCHS:
  60
- ▶
  parallel:{} 2 keys
  - pipeline:
    1
  - ▶
    tensor:{} 3 keys
    - depth:
      1
    - mode:
      "2.5d"
    - size:
      16
- SEQ_LENGTH:
  1,024
- TENSOR_PARALLEL_MODE:
  "2.5d"
- TENSOR_PARALLEL_SIZE:
  16
- TOTAL_BATCH_SIZE:
  1,280
- VOCAB_SIZE:
  50,304
- WARMUP_EPOCHS:
  21
- WEIGHT_DECAY:
  0.01

Summary metrics are your model's outputs. Learn more

▶
Summary metrics:{} 1 key
- hello_message:
  "Hi, we got started"