levmckinney

Levmckinney's group workspace

Group: llama-7b

1-2

of 2

Tags

Notes

Author

levmckinney

State

Finished

Start time

May 7th, 2023 6:30:07 PM

Runtime

2h 51m 51s

Tracked hours

2h 51m 41s

Run path

levmckinney/tuned-lens/y29i9n4d

Linux-5.4.0-148-generic-x86_64-with-glibc2.35

Python version

3.10.6

Command

-m tuned_lens.__main__ train --model.name huggyllama/llama-7b --data.name /datasets/val.jsonl --per_gpu_batch_size=1 --output /output/huggyllama/llama-7b-1683483775 --slow_tokenizer --wandb huggyllama/llama-7b-1683483775 --fsdp

System Hardware

CPU count	128
Logical CPU count	255
GPU count	4
GPU type	NVIDIA A100-SXM4-80GB

W&B CLI Version

0.15.2

Group

llama-7b

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 17 keys
- constant:
  false
- ▶
  data:{} 5 keys
- ▶
  dist:{} 2 keys
  - cpu_offload:
    false
  - fsdp:
    true
- lens_name_or_path:
  null
- loss:
  "LossChoice.KL"
- ▶
  model:{} 5 keys
- num_steps:
  250
- ▶
  opt:{} 6 keys
- output:
  "/output/huggyllama/llama-7b-1683483775"
- per_gpu_batch_size:
  1
- pre_ln:
  false
- seed:
  42
- separate_unembeddings:
  false
- token_shift:
  null
- tokens_per_step:
  262,144
- wandb:
  "huggyllama/llama-7b-1683483775"
- wandb_upload_checkpoints:
  false

Summary metrics are your model's outputs. Learn more

▶
Summary metrics:{} 96 keys
- bias_norm/0.ffn:
  0.549409031867981
- bias_norm/1.ffn:
  0.4864175021648407
- bias_norm/2.ffn:
  0.3815529644489288
- bias_norm/3.ffn:
  0.337003618478775
- bias_norm/4.ffn:
  0.33466318249702454
- bias_norm/5.ffn:
  0.2772126793861389
- bias_norm/6.ffn:
  0.2483600378036499
- bias_norm/7.ffn:
  0.23252029716968536
- bias_norm/8.ffn:
  0.2142762988805771
- bias_norm/9.ffn:
  0.20198982954025269
- bias_norm/10.ffn:
  0.17954114079475403
- bias_norm/11.ffn:
  0.17352469265460968
- bias_norm/12.ffn:
  0.15591587126255035
- bias_norm/13.ffn:
  0.13548873364925385
- bias_norm/14.ffn:
  0.12539741396903992
- bias_norm/15.ffn:
  0.11465544998645782
- bias_norm/16.ffn:
  0.10790852457284927
- bias_norm/17.ffn:
  0.10376321524381638
- bias_norm/18.ffn:
  0.09996820986270905
- bias_norm/19.ffn:
  0.09624125808477402
- bias_norm/20.ffn:
  0.09406394511461258
- bias_norm/21.ffn:
  0.09122977405786514
- bias_norm/22.ffn:
  0.0866076871752739
- bias_norm/23.ffn:
  0.0783386379480362
- bias_norm/24.ffn:
  0.06940832734107971
- bias_norm/25.ffn:
  0.05777710676193237
- bias_norm/26.ffn:
  0.04815864935517311
- bias_norm/27.ffn:
  0.03800032287836075
- bias_norm/28.ffn:
  0.025014115497469906
- bias_norm/29.ffn:
  0.016188008710741997
- bias_norm/30.ffn:
  0.010119954124093056
- bias_norm/input:
  1.4453942775726318
- loss/translator_0:
  1.9336254596710205
- loss/translator_1:
  1.7589058876037598
- loss/translator_2:
  1.6618731021881104
- loss/translator_3:
  1.5882800817489624
- loss/translator_4:
  1.5341249704360962
- loss/translator_5:
  1.5004748106002808
- loss/translator_6:
  1.3935556411743164
- loss/translator_7:
  1.390508770942688
- loss/translator_8:
  1.3534235954284668
- loss/translator_9:
  1.3129788637161257
- loss/translator_10:
  1.262107253074646
- loss/translator_11:
  1.259196162223816
- loss/translator_12:
  1.23857319355011
- loss/translator_13:
  1.1651469469070437
- weight_norm/28.ffn:
  1.7927353382110596
- weight_norm/29.ffn:
  1.5187828540802002
- weight_norm/30.ffn:
  1.2240056991577148
- weight_norm/input:
  3.041059732437134

This run consumed these artifacts as inputs. Learn more

job

job-.io_levmck_hofvarpnir-jobs_sha256_1697a6f5d51ed9a3c08e70e24f73160d68a8fd657031e7db954158118a137073:v0

This run produced these artifacts as outputs. Total: 1. Learn more

wandb-history

run-y29i9n4d-history:v0