collingray

Collingray's group workspace

Group: gpt2_wikitext

1-3

of 3

Tags

Notes

Author

collingray

State

Finished

Start time

April 16th, 2024 5:16:57 AM

Runtime

6h 3m

Tracked hours

6h 2m 56s

Run path

collingray/multilayer_sae/bt6hfsxr

Linux-6.7.8-arch1-1-x86_64-with-glibc2.39

Python version

3.11.4

Git repository

git clone https://github.com/collingray/autoencoders

Git state

git checkout -b "gpt2_wikitext_16x_v2" ef37789752872ebbd07aff953d00d5815dd7f40a

Command

multilayer_experiments.ipynb

System Hardware

CPU count	6
Logical CPU count	12
GPU count	1
GPU type	NVIDIA GeForce RTX 4090

W&B CLI Version

0.16.4

Group

gpt2_wikitext

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 15 keys
- ▶
  act_norms:[] 12 items
- act_renorm_scale:
  1
- act_renorm_type:
  "log"
- device:
  "cuda"
- dtype:
  "torch.bfloat16"
- firing_bucket_size:
  262,144
- lambda_reg:
  0.04
- m_dim:
  12,288
- n_dim:
  768
- name:
  "autoencoder"
- num_firing_buckets:
  16
- record_data:
  true
- save_dir:
  "./weights"
- seed:
  42
- tied:
  false

Summary metrics are your model's outputs. Learn more

▶
Summary metrics:{} 89 keys
- avg_fvu:
  0.003954166546463966
- avg_l0:
  141.46575927734375
- ▶
  feature_density.bins:[] 65 items
- ▶
  feature_density.values:[] 64 items
- layer_0.avg_fvu:
  0.0012690711300820112
- layer_0.avg_l0:
  94.98738098144533
- ▶
  layer_0.feature_density.bins:[] 65 items
- ▶
  layer_0.feature_density.values:[] 64 items
- layer_0.l1:
  0.765625
- layer_0.loss:
  0.03466796875
- layer_0.mse:
  0.00396728515625
- layer_1.avg_fvu:
  0.004373414441943169
- layer_1.avg_l0:
  142.98341369628906
- ▶
  layer_1.feature_density.bins:[] 65 items
- ▶
  layer_1.feature_density.values:[] 64 items
- layer_1.l1:
  1.3046875
- layer_1.loss:
  0.06591796875
- layer_1.mse:
  0.013671875
- layer_2.avg_fvu:
  0.0037291168700903654
- layer_2.avg_l0:
  149.3677520751953
- ▶
  layer_2.feature_density.bins:[] 65 items
- ▶
  layer_2.feature_density.values:[] 64 items
- layer_2.l1:
  2.140625
- layer_2.loss:
  0.09716796875
- layer_2.mse:
  0.01165771484375
- layer_3.avg_fvu:
  0.004041503649204969
- layer_3.avg_l0:
  146.2940673828125
- ▶
  layer_3.feature_density.bins:[] 65 items
- ▶
  layer_3.feature_density.values:[] 64 items
- layer_3.l1:
  1.484375
- layer_3.loss:
  0.07177734375
- layer_3.mse:
  0.01263427734375
- layer_4.avg_fvu:
  0.0033776815980672836
- layer_4.avg_l0:
  148.79832458496094
- ▶
  layer_4.feature_density.bins:[] 65 items
- ▶
  layer_4.feature_density.values:[] 64 items
- layer_4.l1:
  1.4375
- layer_4.loss:
  0.068359375
- layer_4.mse:
  0.01055908203125
- layer_5.avg_fvu:
  0.0034362541045993567
- layer_5.avg_l0:
  147.4556121826172
- ▶
  layer_5.feature_density.bins:[] 65 items
- ▶
  layer_5.feature_density.values:[] 64 items
- layer_5.l1:
  1.0703125
- layer_5.loss:
  0.053466796875
- layer_5.mse:
  0.0107421875
- layer_11.l1:
  0.6640625
- layer_11.loss:
  0.02978515625
- layer_11.mse:
  0.00311279296875
- lr:
  0.00000004898400871955

This run produced these artifacts as outputs. Total: 1. Learn more

wandb-history

run-bt6hfsxr-history:v0