kastan

Kastan's group workspace

Group: Oct-04__16:23

1-2

of 2

Tags

BATCH_SIZE4

NUM_EPOCHS=2

Oct-04__16:23

SLURM=958249

WORLD_SIZE=2

Author

kastan

State

Finished

Start time

October 4th, 2022 9:25:37 PM

Runtime

10m 15s

Tracked hours

10m 9s

Run path

kastan/LLM-Distributed-Quantization/o3znmfuz

Linux-4.18.0-305.57.1.el8_4.x86_64-x86_64-with-glibc2.28

Python version

3.9.12

Command

/u/kastanday/LLM-Distributed-Quantization/benchmarks/gpt/v2_train.py --config /u/kastanday/LLM-Distributed-Quantization/benchmarks/gpt/configs/quant_gpt2_single_gpu.py --host gpub017 --port 29500 --world_size 2 --rank 0

System Hardware

CPU count	64
GPU count	2
GPU type	NVIDIA A40

W&B CLI Version

0.13.0

Group

Oct-04__16:23

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 18 keys
- BATCH_SIZE:
  4
- conda_env_name:
  "col_ai_quant"
- data_dir:
  "/u/kastanday/LLM-Distributed-Quantization/datasets/small-gpt-dataset.json"
- LEARNING_RATE:
  0.00015
- LOG_PATH:
  "./gpt2_bs4_lr0.00015/"
- ▶
  model:{} 7 keys
- ▶
  model_dtypes:{} 4 keys
  - decoder_dtype:
    "torch.bfloat16"
  - embed_dtype:
    "torch.bfloat16"
  - head_dtype:
    "torch.bfloat16"
  - layernorm_dtype:
    "torch.bfloat16"
- NUM_EPOCHS:
  2
- num_gpus_per_node:
  "2"
- ▶
  optimizer:{} 2 keys
  - lr:
    0.00015
  - weight_decay:
    0.01
- ▶
  parallel:{} 1 key
  - pipeline:
    1
- quant_gpt2_small:
  "titans.model.quant_gpt.quant_gpt.quant_gpt2_small"
- SEQ_LENGTH:
  256
- TOTAL_BATCH_SIZE:
  4
- total_gpus:
  "2"
- VOCAB_SIZE:
  50,304
- WARMUP_EPOCHS:
  1
- WEIGHT_DECAY:
  0.01

Summary metrics are your model's outputs. Learn more

▶
Summary metrics:{} 5 keys
- after_train:
  "Goodbye, we finished training"
- embed_layer_norm:
  0
- loss:
  40
- per_step_metrics.samples_per_sec:
  215.12
- per_step_metrics.Tflops:
  24.82

This run produced these artifacts as outputs. Total: 2. Learn more

wandb-history

run-o3znmfuz-history:v0

wandb-events

run-o3znmfuz-events:v0