eleutherai

Igoro's group workspace

Group: 20B_eval_multi

1-13

of 13

Tags

Notes

Author

sdtblck

State

Crashed

Start time

December 24th, 2021 3:35:00 PM

Runtime

21d 4h 55m 19s

Tracked hours

21d 1h 5m 15s

Run path

eleutherai/gpt-thicc/p79m8ebr

Linux-5.8.0-59-generic-x86_64-with-glibc2.29

Python version

3.8.10

Git repository

git clone https://github.com/EleutherAI/gpt-neox.git

Git state

git checkout -b "20B_eval_multi" 3ad61952c290669d3741c01f767d41fdee5215c5

Command

eval_multi.py

System Hardware

CPU count	40
GPU count	8
GPU type	NVIDIA GeForce RTX 2080 Ti

W&B CLI Version

0.10.28

Group

20B_eval_multi

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 53 keys
- attention-dropout:
  0
- bias-gelu-fusion:
  true
- checkpoint-activations:
  true
- checkpoint-num-layers:
  1
- data-impl:
  "mmap"
- data-path:
  "/mnt/ssd-1/data/pile_20B_tokenizer/pile_20B_tokenizer_text_document"
- distributed-backend:
  "nccl"
- eval-interval:
  1,000
- eval-iters:
  10
- ▶
  fp16:{} 7 keys
- gpt_j_residual:
  true
- gradient_accumulation_steps:
  32
- gradient_clipping:
  1
- hidden-dropout:
  0
- hidden-size:
  6,144
- init_method:
  "small_init"
- load:
  "/mnt/checkpoints"
- log-dir:
  "/mnt/ssd-1/logs"
- log-interval:
  2
- lr-decay-iters:
  150,000
- lr-decay-style:
  "cosine"
- max-position-embeddings:
  2,048
- min_lr:
  0.0000097
- model-parallel-size:
  2
- no-weight-tying:
  true
- norm:
  "layernorm"
- num-attention-heads:
  64
- num-layers:
  44
- ▶
  optimizer:{} 2 keys
  - ▶
    params:{} 3 keys
    - ▶
      betas:[] 2 items
      - 0:
        0.9
      - 1:
        0.95
    - eps:
      0.00000001
    - lr:
      0.000097
  - type:
    "Adam"
- output_layer_init_method:
  "wang_init"
- output_layer_parallelism:
  "column"
- partition-activations:
  false
- pipe-parallel-size:
  4
- pos-emb:
  "rotary"
- rotary_pct:
  0.25
- save:
  "/mnt/checkpoints"
- save-interval:
  500
- scaled-upper-triang-masked-softmax-fusion:
  true
- seq-length:
  2,048
- split:
  "995,4,1"
- steps_per_print:
  2
- synchronize-each-layer:
  true
- tensorboard-dir:
  "/mnt/ssd-1/tensorboard"
- tokenizer_type:
  "HFTokenizer"
- train_micro_batch_size_per_gpu:
  4
- train-iters:
  150,000
- wandb_team:
  "eleutherai"
- warmup:
  0.01
- weight-decay:
  0.01
- ▶
  zero_optimization:{} 8 keys

Summary metrics are your model's outputs. Learn more

▶
Summary metrics:{} 20 keys
- validation/hellaswag_acc:
  0.5118502290380402
- validation/hellaswag_acc_norm:
  0.6738697470623382
- validation/hellaswag_acc_norm_stderr:
  0.004678375103797972
- validation/hellaswag_acc_stderr:
  0.0049883798052611636
- validation/lambada_acc:
  0.7098777411216767
- validation/lambada_acc_stderr:
  0.006322580641394927
- validation/lambada_ppl:
  4.4962513479216835
- validation/lambada_ppl_stderr:
  0.17502862429454424
- validation/mathqa_acc:
  0.2693467336683417
- validation/mathqa_acc_norm:
  0.26867671691792294
- validation/mathqa_acc_norm_stderr:
  0.008114659613200549
- validation/mathqa_acc_stderr:
  0.008121048652111622
- validation/piqa_acc:
  0.719804134929271
- validation/piqa_acc_norm:
  0.719804134929271
- validation/piqa_acc_norm_stderr:
  0.010478122015577086
- validation/piqa_acc_stderr:
  0.010478122015577074
- validation/pubmedqa_acc:
  0.663
- validation/pubmedqa_acc_stderr:
  0.014955087918653596
- validation/winogrande_acc:
  0.6085240726124704
- validation/winogrande_acc_stderr:
  0.013717487071290856

This run produced these artifacts as outputs. Total: 1. Learn more

wandb-history

run-p79m8ebr-history:v0