amr-amr

Amr-amr's group workspace

Group: 144M

1-3

of 3

Tags

Main Results (OLMo)

Author

amr-amr

State

Finished

Start time

October 31st, 2024 7:32:16 PM

Runtime

20h 1m 55s

Tracked hours

20h 1m 53s

Run path

amr-amr/zsl/7y3rjwwz

Linux-5.15.0-101-generic-x86_64-with-glibc2.35

Python version

3.10.14

Git repository

git clone git@github.com:mirandrom/zsl-olmo.git

Git state

git checkout -b "1028-rmsnorm-144m.5535327" a5d85f50b6398187a0f97353c8dff998342638b3

Command

/home/mila/m/mirceara/proj/zsl-olmo/scripts/train.py /home/mila/m/mirceara/proj/zsl-olmo/zsl/exp/1028-rmsnorm/144m.yaml --run_name=1028-rmsnorm-144m --wandb.name=1028-rmsnorm-144m.5535327 --wandb.group=1028-rmsnorm-144m --wandb.project=zsl-test --wandb.entity=amr-amr

System Hardware

CPU count	48
Logical CPU count	48
GPU count	2
GPU type	[NVIDIA L40S, NVIDIA L40S]

W&B CLI Version

0.18.1

Group

144M

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 64 keys
- activation_checkpointing:
  null
- auxiliary_loss_multiplier:
  0.0001
- canceled_check_interval:
  50
- ▶
  compile:{} 3 keys
  - backend:
    "inductor"
  - fullgraph:
    false
  - mode:
    "default"
- console_log_interval:
  1
- ▶
  data:{} 15 keys
- ddp:
  null
- device_eval_batch_size:
  16
- device_train_batch_size:
  32
- device_train_grad_accum:
  1
- device_train_microbatch_size:
  32
- distributed_strategy:
  "fsdp"
- dry_run:
  false
- early_stopping_factor:
  null
- epoch:
  null
- eval_interval:
  1,000
- eval_on_load:
  false
- eval_subset_num_batches:
  -1
- evaluators:[] 0 items
- extra_steps_after_cancel:
  10
- fast_forward_batches:
  null
- force_save_unsharded:
  false
- ▶
  fsdp:{} 5 keys
- fused_loss:
  null
- gen1_gc_interval:
  1
- global_train_batch_size:
  512
- hf_datasets_cache_dir:
  null
- load_path:
  null
- load_path_sharded_checkpointer:
  null
- max_duration:
  262,144
- max_grad_norm:
  1
- max_grad_norm_ratio:
  null
- ▶
  model:{} 43 keys
- module_outputs_save_steps:
  null
- new_style_checkpoints:
  null
- no_pre_train_checkpoint:
  false
- ▶
  optimizer:{} 11 keys
- precision:
  "amp_bf16"
- python_profiling:
  false
- remote_save_folder:
  null
- reset_optimizer_state:
  false
- reset_trainer_state:
  false
- restore_dataloader:
  true
- run_name:
  "1028-rmsnorm-144m"
- save_data_indices:
  true
- save_folder:
  "/network/scratch/m/mirceara/zsl-olmo/out/1028-rmsnorm-144m"
- time_limit:
  null
- ▶
  tokenizer:{} 2 keys
  - identifier:
    "tokenizers/allenai_eleuther-ai-gpt-neox-20b-pii-special.json"
  - truncate_direction:
    "right"
- torch_profiling:
  false
- try_load_latest_save:
  true

Summary metrics are your model's outputs. Learn more

▶
Summary metrics:{} 1052 keys
- optim/clipping_rate:
  0
- optim/exp_avg_sq/transformer.blocks.0.att_proj.weight.avg:
  0.00000000140815770155
- optim/exp_avg_sq/transformer.blocks.0.att_proj.weight.max:
  0.00000374880141862377
- optim/exp_avg_sq/transformer.blocks.0.att_proj.weight.min:
  0.00000000000649515207
- optim/exp_avg_sq/transformer.blocks.0.att_proj.weight.norm:
  0.00003821092104772106
- optim/exp_avg_sq/transformer.blocks.0.attn_out.weight.avg:
  0.00000000324503446336
- optim/exp_avg_sq/transformer.blocks.0.attn_out.weight.max:
  0.00000395380948248203
- optim/exp_avg_sq/transformer.blocks.0.attn_out.weight.min:
  0.00000000004517177604
- optim/exp_avg_sq/transformer.blocks.0.attn_out.weight.norm:
  0.00003113623461103998
- optim/exp_avg_sq/transformer.blocks.0.ff_out.weight.avg:
  0.00000000050047321842
- optim/exp_avg_sq/transformer.blocks.0.ff_out.weight.max:
  0.00000013935303400103
- optim/exp_avg_sq/transformer.blocks.0.ff_out.weight.min:
  0.00000000005873927039
- optim/exp_avg_sq/transformer.blocks.0.ff_out.weight.norm:
  0.0000007163507120822
- optim/exp_avg_sq/transformer.blocks.0.ff_proj.weight.avg:
  0.00000000070730027391
- optim/exp_avg_sq/transformer.blocks.0.ff_proj.weight.max:
  0.00001100437839340884
- optim/exp_avg_sq/transformer.blocks.0.ff_proj.weight.min:
  0.00000000001395574021
- optim/exp_avg_sq/transformer.blocks.0.ff_proj.weight.norm:
  0.00003077584915445186
- optim/exp_avg_sq/transformer.blocks.1.att_proj.weight.avg:
  0.00000000232089236718
- optim/exp_avg_sq/transformer.blocks.1.att_proj.weight.max:
  0.00013640183897223324
- optim/exp_avg_sq/transformer.blocks.1.att_proj.weight.min:
  0.00000000000088529927
- optim/exp_avg_sq/transformer.blocks.1.att_proj.weight.norm:
  0.00031213246984407306
- optim/exp_avg_sq/transformer.blocks.1.attn_out.weight.avg:
  0.00000000028370422678
- optim/exp_avg_sq/transformer.blocks.1.attn_out.weight.max:
  0.00000019409800700032
- optim/exp_avg_sq/transformer.blocks.1.attn_out.weight.min:
  0.00000000002227325269
- optim/exp_avg_sq/transformer.blocks.1.attn_out.weight.norm:
  0.00000074548904649419
- optim/exp_avg_sq/transformer.blocks.1.ff_out.weight.avg:
  0.00000000032114827464
- optim/exp_avg_sq/transformer.blocks.1.ff_out.weight.max:
  0.0000000735281417974
- optim/exp_avg_sq/transformer.blocks.1.ff_out.weight.min:
  0.00000000005387880767
- optim/exp_avg_sq/transformer.blocks.1.ff_out.weight.norm:
  0.00000045065695530866
- optim/exp_avg_sq/transformer.blocks.1.ff_proj.weight.avg:
  0.00000000067187932995
- optim/exp_avg_sq/transformer.blocks.1.ff_proj.weight.max:
  0.00000923082188819535
- optim/exp_avg_sq/transformer.blocks.1.ff_proj.weight.min:
  0.00000000002816543179
- optim/exp_avg_sq/transformer.blocks.1.ff_proj.weight.norm:
  0.00002711828710744157
- optim/exp_avg_sq/transformer.blocks.2.att_proj.weight.avg:
  0.00000000247471887427
- optim/exp_avg_sq/transformer.blocks.2.att_proj.weight.max:
  0.00005588836211245507
- optim/exp_avg_sq/transformer.blocks.2.att_proj.weight.min:
  0.00000000000181332575
- optim/exp_avg_sq/transformer.blocks.2.att_proj.weight.norm:
  0.00028939652838744223
- optim/exp_avg_sq/transformer.blocks.2.attn_out.weight.avg:
  0.00000000027762123156
- optim/exp_avg_sq/transformer.blocks.2.attn_out.weight.max:
  0.00000006437569055606
- optim/exp_avg_sq/transformer.blocks.2.attn_out.weight.min:
  0.00000000003545549271
- optim/exp_avg_sq/transformer.blocks.2.attn_out.weight.norm:
  0.00000058846177353189
- optim/exp_avg_sq/transformer.blocks.2.ff_out.weight.avg:
  0.00000003701269335465
- optim/exp_avg_sq/transformer.blocks.2.ff_out.weight.max:
  0.00011910068860743196
- optim/exp_avg_sq/transformer.blocks.2.ff_out.weight.min:
  0.00000000005311029047
- optim/exp_avg_sq/transformer.blocks.2.ff_out.weight.norm:
  0.0006942764157429338
- optim/exp_avg_sq/transformer.blocks.2.ff_proj.weight.avg:
  0.00000000070682132369
- throughput/total_training_Gflops:
  34,197,136,438.091316
- throughput/total_training_log_Gflops:
  24.255407747629057
- train/CrossEntropyLoss:
  3.033912181854248
- train/Perplexity:
  20.77836252120684

This run consumed these artifacts as inputs. Learn more

No rows found

This run produced these artifacts as outputs. Total: 1. Learn more

wandb-history

run-7y3rjwwz-history:v1