gpt2_zero3 | col_ai Workspace – Weights & Biases

Skip to main content

Kastan's group workspace

Group: gpt2_zero3

24

1-1

of 1

Tags

Notes

Author

State

Failed

Start time

June 22nd, 2022 4:32:19 PM

Runtime

39s

Tracked hours

36s

Run path

kastan/col_ai/u34l1jlo

OS

Linux-4.18.0-348.23.1.el8_5.x86_64-x86_64-with-glibc2.10

Python version

3.8.2

Git repository

git clone https://github.com/hpcaitech/ColossalAI-Examples.git

Git state

git checkout -b "gpt2_zero3" 0837bc67d4d7032c6c29f18339aef6dff6c7b861

Command

train_gpt.py --config gpt2_configs/gpt2_zero3.py --from_torch

System Hardware

CPU count	128
GPU count	4
GPU type	NVIDIA A100-SXM4-80GB

W&B CLI Version

0.12.18

Group

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 17 keys
- backend:
  "nccl"
- BATCH_SIZE:
  2
- colossal_config_file:
  "gpt2_configs/gpt2_zero3.py"
- config:
  "gpt2_configs/gpt2_zero3.py"
- data_dir:
  "/u/kastan/colossal/data/train_data_FINAL.json"
- from_torch:
  true
- gpt2_small:
  "titans.model.gpt.gpt.gpt2_small"
- host:
  null
- local_rank:
  null
- ▶
  model:{} 2 keys
  - checkpoint:
    true
  - type:
    "titans.model.gpt.gpt.gpt2_small"
- NUM_EPOCHS:
  60
- ▶
  optimizer:{} 3 keys
  - lr:
    0.00015
  - type:
    "colossalai.nn.optimizer.hybrid_adam.HybridAdam"
  - weight_decay:
    0.01
- port:
  null
- rank:
  null
- SEQ_LEN:
  1,024
- world_size:
  null
- ▶
  zero:{} 2 keys
  - ▶
    model_config:{} 3 keys
    - reuse_fp16_shard:
      true
    - shard_strategy:
      "<colossalai.zero.shard_utils.tensor_shard_strategy.TensorShardStrategy object at 0x150e7506b340>"
    - tensor_placement_policy:
      "cpu"
  - optimizer_config:{} 0 keys

Summary metrics are your model's outputs. Learn more

▶
Summary metrics:{} 5 keys
- hello_message:
  "Hi, we got started"
- loss:
  82.625
- per_step_metrics.loss:
  82.625
- per_step_metrics.samples_per_sec:
  61.968
- per_step_metrics.Tflops:
  14.367

This run produced these artifacts as outputs. Total: 1. Learn more

run-u34l1jlo-history:v0

Loading...