kastan

Kastan's group workspace

Group: Oct-04__16:06

1-1

of 1

Tags

BATCH_SIZE4

NUM_EPOCHS=2

Oct-04__16:06

SLURM=958216

WORLD_SIZE=1

Author

kastan

State

Finished

Start time

October 4th, 2022 9:09:25 PM

Runtime

7m 1s

Tracked hours

6m 40s

Run path

kastan/LLM-Distributed-Quantization/3khqn45o

Linux-4.18.0-305.57.1.el8_4.x86_64-x86_64-with-glibc2.28

Python version

3.9.12

Command

/u/kastanday/LLM-Distributed-Quantization/benchmarks/gpt/v2_train.py --config /u/kastanday/LLM-Distributed-Quantization/benchmarks/gpt/configs/quant_gpt2_single_gpu.py --host gpub075 --port 29500 --world_size 1 --rank 0

System Hardware

CPU count	64
GPU count	1
GPU type	NVIDIA A40

W&B CLI Version

0.13.0

Group

Oct-04__16:06

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 18 keys
- BATCH_SIZE:
  4
- conda_env_name:
  "col_ai_quant"
- data_dir:
  "/u/kastanday/LLM-Distributed-Quantization/datasets/small-gpt-dataset.json"
- LEARNING_RATE:
  0.00015
- LOG_PATH:
  "./gpt2_bs4_lr0.00015/"
- ▶
  model:{} 7 keys
- ▶
  model_dtypes:{} 4 keys
  - decoder_dtype:
    "torch.bfloat16"
  - embed_dtype:
    "torch.bfloat16"
  - head_dtype:
    "torch.bfloat16"
  - layernorm_dtype:
    "torch.bfloat16"
- NUM_EPOCHS:
  2
- num_gpus_per_node:
  "1"
- ▶
  optimizer:{} 2 keys
  - lr:
    0.00015
  - weight_decay:
    0.01
- ▶
  parallel:{} 1 key
  - pipeline:
    1
- quant_gpt2_small:
  "titans.model.quant_gpt.quant_gpt.quant_gpt2_small"
- SEQ_LENGTH:
  256
- TOTAL_BATCH_SIZE:
  4
- total_gpus:
  "1"
- VOCAB_SIZE:
  50,304
- WARMUP_EPOCHS:
  1
- WEIGHT_DECAY:
  0.01

Summary metrics are your model's outputs. Learn more

▶
Summary metrics:{} 6 keys
- after_train:
  "Goodbye, we finished training"
- embed_layer_norm:
  1.5242896676063538
- head_layer_norm:
  1.523681640625
- loss:
  40.5
- per_step_metrics.samples_per_sec:
  101.79
- per_step_metrics.Tflops:
  23.489

This run produced these artifacts as outputs. Total: 2. Learn more

wandb-history

run-3khqn45o-history:v0

wandb-events

run-3khqn45o-events:v0