kastan

Kastan's group workspace

Group: Aug-05__13:58

1-16

of 16

Tags

Aug-05__13:58

BATCH_SIZE32

MICRO_BATCH_SIZE=4

NUM_EPOCHS=20

NUM_MICRO_BATCHES=8

PP=2

SLURM=513923

TP=4

WORLD_SIZE=16

Author

kastan

State

Crashed

Start time

August 5th, 2022 6:59:45 PM

Runtime

28m 3s

Tracked hours

Run path

kastan/LLM-Distributed-Quantization/3nsgkzdv

Linux-4.18.0-305.49.1.el8_4.x86_64-x86_64-with-glibc2.28

Python version

3.9.12

Command

/u/kastanday/LLM-Distributed-Quantization/benchmarks/gpt/v2_train.py --config /u/kastanday/LLM-Distributed-Quantization/benchmarks/gpt/configs/gpt2_PP2_TP4_2d.py --host gpub007 --port 29500 --world_size 16 --rank 6

System Hardware

CPU count	64
GPU count	4
GPU type	NVIDIA A40

W&B CLI Version

0.13.0

Group

Aug-05__13:58

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 26 keys
- BATCH_SIZE:
  32
- clip_grad_norm:
  1
- conda_env_name:
  "col_ai_quant"
- data_dir:
  "/u/kastanday/LLM-Distributed-Quantization/datasets/small-gpt-dataset.json"
- ▶
  fp16:{} 1 key
  - mode:
    "AMP_TYPE.NAIVE"
- gpt2_medium:
  "titans.model.gpt.gpt.gpt2_medium"
- gradient_accumulation:
  1
- LEARNING_RATE:
  0.00015
- ▶
  loss:{} 1 key
  - type:
    "titans.loss.lm_loss.gpt_lmloss.GPTLMLoss"
- MICRO_BATCH_SIZE:
  4
- ▶
  model:{} 4 keys
  - checkpoint:
    true
  - dtype:
    "torch.float16"
  - max_position_embeddings:
    1,024
  - vocab_size:
    50,304
- NUM_EPOCHS:
  20
- num_gpus_per_node:
  "4"
- NUM_MICRO_BATCHES:
  8
- ▶
  optimizer:{} 2 keys
  - lr:
    0.00015
  - weight_decay:
    0.01
- ▶
  parallel:{} 2 keys
  - pipeline:
    2
  - ▶
    tensor:{} 2 keys
    - mode:
      "2d"
    - size:
      4
- PIPELINE_SIZE:
  2
- ▶
  schedule:{} 4 keys
  - num_microbatches:
    8
  - scatter_gather_tensors:
    true
  - ▶
    tensor_shape:[] 3 items
    - 0:
      4
    - 1:
      1,024
    - 2:
      3,072
  - type:
    "colossalai.engine.schedule._pipeline_schedule.PipelineSchedule"
- SEQ_LENGTH:
  1,024
- TENSOR_PARALLEL_MODE:
  "2d"
- TENSOR_PARALLEL_SIZE:
  4
- TOTAL_BATCH_SIZE:
  32
- total_gpus:
  "16"
- VOCAB_SIZE:
  50,304
- WARMUP_EPOCHS:
  1
- WEIGHT_DECAY:
  0.01

Summary metrics are your model's outputs. Learn more

No summary metrics saved for this run.

Check the summary metrics documentation for more information.

This run produced these artifacts as outputs. Total: 1. Learn more

wandb-events

run-3nsgkzdv-events:v0