kastan

Kastan's group workspace

Group: Aug-05__10:49

1-2

of 2

Tags

Aug-05__10:49

BATCH_SIZE4

NUM_EPOCHS=2

SLURM=513225

WORLD_SIZE=2

Author

kastan

State

Finished

Start time

August 5th, 2022 3:50:52 PM

Runtime

7m 16s

Tracked hours

7m 12s

Run path

kastan/LLM-Distributed-Quantization/3pibka98

Linux-4.18.0-305.49.1.el8_4.x86_64-x86_64-with-glibc2.28

Python version

3.9.12

Command

/u/kastanday/LLM-Distributed-Quantization/benchmarks/gpt/v2_train.py --config /u/kastanday/LLM-Distributed-Quantization/benchmarks/gpt/configs/quant_gpt2_single_gpu.py --host gpub026 --port 29500 --world_size 2 --rank 1

System Hardware

CPU count	64
GPU count	1
GPU type	NVIDIA A40

W&B CLI Version

0.13.0

Group

Aug-05__10:49

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 15 keys
- BATCH_SIZE:
  4
- data_dir:
  "/u/kastanday/LLM-Distributed-Quantization/datasets/small-gpt-dataset.json"
- LEARNING_RATE:
  0.00015
- LOG_PATH:
  "./gpt2_bs4_lr0.00015/"
- ▶
  model:{} 7 keys
- ▶
  model_dtypes:{} 4 keys
  - decoder_dtype:
    "torch.bfloat16"
  - embed_dtype:
    "torch.bfloat16"
  - head_dtype:
    "torch.bfloat16"
  - layernorm_dtype:
    "torch.bfloat16"
- NUM_EPOCHS:
  2
- ▶
  optimizer:{} 2 keys
  - lr:
    0.00015
  - weight_decay:
    0.01
- ▶
  parallel:{} 1 key
  - pipeline:
    1
- quant_gpt2_small:
  "titans.model.quant_gpt.quant_gpt.quant_gpt2_small"
- SEQ_LENGTH:
  256
- TOTAL_BATCH_SIZE:
  4
- VOCAB_SIZE:
  50,304
- WARMUP_EPOCHS:
  1
- WEIGHT_DECAY:
  0.01

Summary metrics are your model's outputs. Learn more

▶
Summary metrics:{} 5 keys
- after_train:
  "Goodbye, we finished training"
- embed_layer_norm:
  0
- loss:
  41
- per_step_metrics.samples_per_sec:
  213.68
- per_step_metrics.Tflops:
  24.654

This run produced these artifacts as outputs. Total: 1. Learn more

wandb-history

run-3pibka98-history:v0