amr-amr

Amr-amr's group workspace

Group: 78M

1-2

of 2

Tags

Main Results (OLMo)

Author

amr-amr

State

Finished

Start time

October 29th, 2024 9:53:26 PM

Runtime

8h 11m 11s

Tracked hours

8h 11m 10s

Run path

amr-amr/zsl/7t13yq2u

Linux-5.15.0-101-generic-x86_64-with-glibc2.35

Python version

3.10.14

Git repository

git clone git@github.com:mirandrom/zsl-olmo.git

Git state

git checkout -b "1028-rmsnorm-78m" a756ab5509e5fc6cafa09b061cfe042b2d1e9fe4

Command

/home/mila/m/mirceara/proj/zsl-olmo/scripts/train.py /home/mila/m/mirceara/proj/zsl-olmo/zsl/exp/1028-rmsnorm/78m.yaml --run_name=1028-rmsnorm-78m --wandb.name=1028-rmsnorm-78m --wandb.group=1028-rmsnorm-78m --wandb.project=zsl-test --wandb.entity=amr-amr

System Hardware

CPU count	48
Logical CPU count	48
GPU count	2
GPU type	[NVIDIA L40S, NVIDIA L40S]

W&B CLI Version

0.18.1

Group

78M

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 64 keys
- activation_checkpointing:
  null
- auxiliary_loss_multiplier:
  0.0001
- canceled_check_interval:
  50
- ▶
  compile:{} 3 keys
  - backend:
    "inductor"
  - fullgraph:
    false
  - mode:
    "default"
- console_log_interval:
  1
- ▶
  data:{} 15 keys
- ddp:
  null
- device_eval_batch_size:
  16
- device_train_batch_size:
  32
- device_train_grad_accum:
  1
- device_train_microbatch_size:
  32
- distributed_strategy:
  "fsdp"
- dry_run:
  false
- early_stopping_factor:
  null
- epoch:
  null
- eval_interval:
  1,000
- eval_on_load:
  false
- eval_subset_num_batches:
  -1
- evaluators:[] 0 items
- extra_steps_after_cancel:
  10
- fast_forward_batches:
  null
- force_save_unsharded:
  false
- ▶
  fsdp:{} 5 keys
- fused_loss:
  null
- gen1_gc_interval:
  1
- global_train_batch_size:
  512
- hf_datasets_cache_dir:
  null
- load_path:
  null
- load_path_sharded_checkpointer:
  null
- max_duration:
  262,144
- max_grad_norm:
  1
- max_grad_norm_ratio:
  null
- ▶
  model:{} 43 keys
- module_outputs_save_steps:
  null
- new_style_checkpoints:
  null
- no_pre_train_checkpoint:
  false
- ▶
  optimizer:{} 11 keys
- precision:
  "amp_bf16"
- python_profiling:
  false
- remote_save_folder:
  null
- reset_optimizer_state:
  false
- reset_trainer_state:
  false
- restore_dataloader:
  true
- run_name:
  "1028-rmsnorm-78m"
- save_data_indices:
  true
- save_folder:
  "/network/scratch/m/mirceara/zsl-olmo/out/1028-rmsnorm-78m"
- time_limit:
  null
- ▶
  tokenizer:{} 2 keys
  - identifier:
    "tokenizers/allenai_eleuther-ai-gpt-neox-20b-pii-special.json"
  - truncate_direction:
    "right"
- torch_profiling:
  false
- try_load_latest_save:
  true

Summary metrics are your model's outputs. Learn more

▶
Summary metrics:{} 796 keys
- optim/clipping_rate:
  0
- optim/exp_avg_sq/transformer.blocks.0.att_proj.weight.avg:
  0.0000000042812113854
- optim/exp_avg_sq/transformer.blocks.0.att_proj.weight.max:
  0.00001729361974867061
- optim/exp_avg_sq/transformer.blocks.0.att_proj.weight.min:
  0.00000000000969423153
- optim/exp_avg_sq/transformer.blocks.0.att_proj.weight.norm:
  0.00012260410585440695
- optim/exp_avg_sq/transformer.blocks.0.attn_out.weight.avg:
  0.0000000082338509344
- optim/exp_avg_sq/transformer.blocks.0.attn_out.weight.max:
  0.00000583794553676853
- optim/exp_avg_sq/transformer.blocks.0.attn_out.weight.min:
  0.00000000010502651526
- optim/exp_avg_sq/transformer.blocks.0.attn_out.weight.norm:
  0.00004766496931551955
- optim/exp_avg_sq/transformer.blocks.0.ff_out.weight.avg:
  0.00000000186995552376
- optim/exp_avg_sq/transformer.blocks.0.ff_out.weight.max:
  0.00000147871105582453
- optim/exp_avg_sq/transformer.blocks.0.ff_out.weight.min:
  0.00000000012757293832
- optim/exp_avg_sq/transformer.blocks.0.ff_out.weight.norm:
  0.00000433032573710079
- optim/exp_avg_sq/transformer.blocks.0.ff_proj.weight.avg:
  0.00000000171958136619
- optim/exp_avg_sq/transformer.blocks.0.ff_proj.weight.max:
  0.0000225899839279009
- optim/exp_avg_sq/transformer.blocks.0.ff_proj.weight.min:
  0.00000000004731354197
- optim/exp_avg_sq/transformer.blocks.0.ff_proj.weight.norm:
  0.00005257746306597255
- optim/exp_avg_sq/transformer.blocks.1.att_proj.weight.avg:
  0.00000000588923487754
- optim/exp_avg_sq/transformer.blocks.1.att_proj.weight.max:
  0.00010391221439931542
- optim/exp_avg_sq/transformer.blocks.1.att_proj.weight.min:
  0.0000000000075972284
- optim/exp_avg_sq/transformer.blocks.1.att_proj.weight.norm:
  0.0004308274365030229
- optim/exp_avg_sq/transformer.blocks.1.attn_out.weight.avg:
  0.00000000058071242259
- optim/exp_avg_sq/transformer.blocks.1.attn_out.weight.max:
  0.00000028970654852856
- optim/exp_avg_sq/transformer.blocks.1.attn_out.weight.min:
  0.00000000004596043685
- optim/exp_avg_sq/transformer.blocks.1.attn_out.weight.norm:
  0.0000011099323273811
- optim/exp_avg_sq/transformer.blocks.1.ff_out.weight.avg:
  0.00000000065398819693
- optim/exp_avg_sq/transformer.blocks.1.ff_out.weight.max:
  0.0000001208015589782
- optim/exp_avg_sq/transformer.blocks.1.ff_out.weight.min:
  0.00000000010687709051
- optim/exp_avg_sq/transformer.blocks.1.ff_out.weight.norm:
  0.00000064599282723066
- optim/exp_avg_sq/transformer.blocks.1.ff_proj.weight.avg:
  0.00000000177133585577
- optim/exp_avg_sq/transformer.blocks.1.ff_proj.weight.max:
  0.000090082197857555
- optim/exp_avg_sq/transformer.blocks.1.ff_proj.weight.min:
  0.00000000004575950729
- optim/exp_avg_sq/transformer.blocks.1.ff_proj.weight.norm:
  0.000101468940556515
- optim/exp_avg_sq/transformer.blocks.2.att_proj.weight.avg:
  0.00000000745202033414
- optim/exp_avg_sq/transformer.blocks.2.att_proj.weight.max:
  0.0001638083631405607
- optim/exp_avg_sq/transformer.blocks.2.att_proj.weight.min:
  0.00000000000149296894
- optim/exp_avg_sq/transformer.blocks.2.att_proj.weight.norm:
  0.0005756734753958881
- optim/exp_avg_sq/transformer.blocks.2.attn_out.weight.avg:
  0.00000000059414101417
- optim/exp_avg_sq/transformer.blocks.2.attn_out.weight.max:
  0.00000042839280922635
- optim/exp_avg_sq/transformer.blocks.2.attn_out.weight.min:
  0.00000000005508141859
- optim/exp_avg_sq/transformer.blocks.2.attn_out.weight.norm:
  0.00000124257860534271
- optim/exp_avg_sq/transformer.blocks.2.ff_out.weight.avg:
  0.00000000941686906231
- optim/exp_avg_sq/transformer.blocks.2.ff_out.weight.max:
  0.00007480040949303657
- optim/exp_avg_sq/transformer.blocks.2.ff_out.weight.min:
  0.0000000001108010142
- optim/exp_avg_sq/transformer.blocks.2.ff_out.weight.norm:
  0.00012174207222415134
- optim/exp_avg_sq/transformer.blocks.2.ff_proj.weight.avg:
  0.00000000200989691557
- throughput/total_training_Gflops:
  18,230,140,789.84382
- throughput/total_training_log_Gflops:
  23.626342148582687
- train/CrossEntropyLoss:
  3.1946680545806885
- train/Perplexity:
  24.402072191655765

This run consumed these artifacts as inputs. Learn more

No rows found

This run produced these artifacts as outputs. Total: 1. Learn more

wandb-history

run-7t13yq2u-history:v0