gpt2_3d | col_ai Workspace – Weights & Biases

Skip to main content

Kastan's group workspace

Group: gpt2_3d

48

1-1

of 1

Tags

Notes

Tags

delta

gpt2_8B

Author

State

Finished

Start time

June 25th, 2022 4:11:35 AM

Runtime

4m 1s

Tracked hours

3m 59s

Run path

kastan/col_ai/tvn44a8j

OS

Linux-4.18.0-305.49.1.el8_4.x86_64-x86_64-with-glibc2.17

Python version

3.8.13

Git repository

git clone https://github.com/hpcaitech/ColossalAI-Examples.git

Git state

git checkout -b "gpt2_3d" 45b6e9cc24b57aed4c5cf2c9278277e74cdef541

Command

train_gpt.py --config gpt2_configs/gpt2_3d.py --from_torch

System Hardware

CPU count	128
GPU count	8
GPU type	NVIDIA A100-SXM4-40GB

W&B CLI Version

0.12.19

Group

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 23 keys
- backend:
  "nccl"
- BATCH_SIZE:
  4
- colossal_config_file:
  "gpt2_configs/gpt2_3d.py"
- config:
  "gpt2_configs/gpt2_3d.py"
- data_dir:
  "/u/kastanday/colossal_ai/raw_json_backup/train_data_FINAL.json"
- ▶
  fp16:{} 1 key
  - mode:
    "AMP_TYPE.NAIVE"
- from_col:
  false
- from_torch:
  true
- gpt2_8B:
  "titans.model.gpt.gpt.gpt2_8B"
- gpt2_small:
  "titans.model.gpt.gpt.gpt2_small"
- gpt2_xl:
  "titans.model.gpt.gpt.gpt2_xl"
- host:
  null
- local_rank:
  null
- ▶
  loss:{} 1 key
  - type:
    "titans.loss.lm_loss.gpt_lmloss.GPTLMLoss"
- ▶
  model:{} 2 keys
  - checkpoint:
    true
  - type:
    "titans.model.gpt.gpt.gpt2_xl"
- NUM_EPOCHS:
  60
- ▶
  optimizer:{} 3 keys
  - lr:
    0.00015
  - type:
    "torch.optim.adam.Adam"
  - weight_decay:
    0.01
- ▶
  parallel:{} 2 keys
  - pipeline:
    1
  - ▶
    tensor:{} 2 keys
    - mode:
      "3d"
    - size:
      8
- port:
  null
- rank:
  null
- SEQ_LEN:
  1,024
- TENSOR_PARALLEL:
  8
- world_size:
  null

Summary metrics are your model's outputs. Learn more

▶
Summary metrics:{} 2 keys
- hello_message:
  "Hi, we got started"
- loss:
  7.564544677734375

This run produced these artifacts as outputs. Total: 1. Learn more

run-tvn44a8j-history:v0

Loading...