eleutherai

Igoro's group workspace

Group: 20B_eval

1-8

of 8

Tags

Notes

Author

sdtblck

State

Crashed

Start time

November 24th, 2021 2:30:18 AM

Runtime

20h 39m 29s

Tracked hours

Run path

eleutherai/gpt-thicc/1pg2z8jn

Linux-5.11.0-40-generic-x86_64-with-glibc2.10

Python version

3.8.5

Git repository

git clone https://github.com/EleutherAI/pyfra_scripts.git

Git state

git checkout -b "20B_eval" 36bdc8af77fc204c06e375cc834929a590bed8f3

Command

train_model.py

System Hardware

CPU count	12
GPU count	1
GPU type	GeForce RTX 3080

W&B CLI Version

0.12.1

Group

20B_eval

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 53 keys
- attention-dropout:
  0
- bias-gelu-fusion:
  true
- checkpoint-activations:
  true
- checkpoint-num-layers:
  1
- data-impl:
  "mmap"
- data-path:
  "/mnt/ssd-1/data/pile_20B_tokenizer/pile_20B_tokenizer_text_document"
- distributed-backend:
  "nccl"
- eval-interval:
  1,000
- eval-iters:
  10
- ▶
  fp16:{} 7 keys
- gpt_j_residual:
  true
- gradient_accumulation_steps:
  32
- gradient_clipping:
  1
- hidden-dropout:
  0
- hidden-size:
  6,144
- init_method:
  "small_init"
- load:
  "/mnt/ssd-1/20B_checkpoints"
- log-dir:
  "/mnt/ssd-1/logs"
- log-interval:
  2
- lr-decay-iters:
  150,000
- lr-decay-style:
  "cosine"
- max-position-embeddings:
  2,048
- min_lr:
  0.0000097
- model-parallel-size:
  2
- no-weight-tying:
  true
- norm:
  "layernorm"
- num-attention-heads:
  64
- num-layers:
  44
- ▶
  optimizer:{} 2 keys
  - ▶
    params:{} 3 keys
    - ▶
      betas:[] 2 items
      - 0:
        0.9
      - 1:
        0.95
    - eps:
      0.00000001
    - lr:
      0.000097
  - type:
    "Adam"
- output_layer_init_method:
  "wang_init"
- output_layer_parallelism:
  "column"
- partition-activations:
  false
- pipe-parallel-size:
  4
- pos-emb:
  "rotary"
- rotary_pct:
  0.25
- save:
  "/mnt/ssd-1/20B_checkpoints"
- save-interval:
  1,000
- scaled-upper-triang-masked-softmax-fusion:
  true
- seq-length:
  2,048
- split:
  "995,4,1"
- steps_per_print:
  2
- synchronize-each-layer:
  true
- tensorboard-dir:
  "/mnt/ssd-1/tensorboard"
- tokenizer_type:
  "HFTokenizer"
- train_micro_batch_size_per_gpu:
  4
- train-iters:
  150,000
- wandb_team:
  "eleutherai"
- warmup:
  0.01
- weight-decay:
  0.01
- ▶
  zero_optimization:{} 8 keys

Summary metrics are your model's outputs. Learn more

No summary metrics saved for this run.

Check the summary metrics documentation for more information.