kastan

Kastan's group workspace

Group: Aug-05__14:16

1-20

of 32

Tags

Aug-05__14:16

BATCH_SIZE1280

NUM_EPOCHS=60

NUM_MICRO_BATCHES=8

SLURM=513930

TP=16

WORLD_SIZE=32

Author

kastan

State

Crashed

Start time

August 5th, 2022 7:18:51 PM

Runtime

27m 37s

Tracked hours

Run path

kastan/LLM-Distributed-Quantization/glggu52z

Linux-4.18.0-305.49.1.el8_4.x86_64-x86_64-with-glibc2.28

Python version

3.9.12

Command

/u/kastanday/LLM-Distributed-Quantization/benchmarks/gpt/v2_train.py --config /u/kastanday/LLM-Distributed-Quantization/benchmarks/gpt/configs/q_allBF16_gpt2_8b_2p5d_256.py --host gpub032 --port 29500 --world_size 32 --rank 9

System Hardware

CPU count	64
GPU count	4
GPU type	NVIDIA A40

W&B CLI Version

0.13.0

Group

Aug-05__14:16

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 24 keys
- BATCH_SIZE:
  1,280
- clip_grad_norm:
  1
- conda_env_name:
  "col_ai_quant"
- data_dir:
  "/u/kastanday/LLM-Distributed-Quantization/datasets/small-gpt-dataset.json"
- gradient_accumulation:
  1
- LEARNING_RATE:
  0.00015
- LOG_PATH:
  "./gpt2_2.5d_tp16_bs1280_lr0.00015_accum1_clip_grad1.0/"
- ▶
  loss:{} 1 key
  - type:
    "titans.loss.lm_loss.gpt_lmloss.GPTLMLoss"
- ▶
  model:{} 8 keys
- ▶
  model_dtypes:{} 4 keys
  - decoder_dtype:
    "torch.bfloat16"
  - embed_dtype:
    "torch.bfloat16"
  - head_dtype:
    "torch.bfloat16"
  - layernorm_dtype:
    "torch.bfloat16"
- NUM_EPOCHS:
  60
- num_gpus_per_node:
  "4"
- NUM_MICRO_BATCHES:
  8
- ▶
  optimizer:{} 2 keys
  - lr:
    0.00015
  - weight_decay:
    0.01
- ▶
  parallel:{} 2 keys
  - pipeline:
    2
  - ▶
    tensor:{} 3 keys
    - depth:
      1
    - mode:
      "2.5d"
    - size:
      16
- quant_gpt2_8B:
  "titans.model.quant_gpt.quant_gpt.quant_gpt2_8B"
- SEQ_LENGTH:
  1,024
- TENSOR_PARALLEL_MODE:
  "2.5d"
- TENSOR_PARALLEL_SIZE:
  16
- TOTAL_BATCH_SIZE:
  1,280
- total_gpus:
  "32"
- VOCAB_SIZE:
  50,304
- WARMUP_EPOCHS:
  21
- WEIGHT_DECAY:
  0.01

Summary metrics are your model's outputs. Learn more

No summary metrics saved for this run.

Check the summary metrics documentation for more information.

This run produced these artifacts as outputs. Total: 1. Learn more

wandb-events

run-glggu52z-events:v0