amr-amr

Amr-amr's group workspace

Group: 472M

1-10

of 10

Tags

Main Results (OLMo)

Author

amr-amr

State

Finished

Start time

April 3rd, 2025 3:28:01 AM

Runtime

1d 20h 3m 59s

Tracked hours

Run path

amr-amr/zsl/0dvyk2gc

Linux-5.15.0-101-generic-x86_64-with-glibc2.35

Python version

CPython 3.10.14

Git repository

git clone git@github.com:mirandrom/zsl-olmo.git

Git state

git checkout -b "1028-rmsnorm-472m.6515011" e5db42f54d08fb0b5412ee6268cd20471c7a7467

Command

/home/mila/m/mirceara/proj/zsl-olmo/scripts/train.py /home/mila/m/mirceara/proj/zsl-olmo/zsl/exp/1028-rmsnorm/472m.yaml --run_name=1028-rmsnorm-472m --wandb.name=1028-rmsnorm-472m.6515011 --wandb.group=1028-rmsnorm-472m --wandb.project=zsl-test --wandb.entity=amr-amr

System Hardware

CPU count	48
Logical CPU count	48
GPU count	2
GPU type	NVIDIA L40S

W&B CLI Version

0.19.8

Group

472M

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 64 keys
- activation_checkpointing:
  "fine_grained"
- auxiliary_loss_multiplier:
  0.0001
- canceled_check_interval:
  50
- ▶
  compile:{} 3 keys
  - backend:
    "inductor"
  - fullgraph:
    false
  - mode:
    "default"
- console_log_interval:
  1
- ▶
  data:{} 16 keys
- ddp:
  null
- device_eval_batch_size:
  16
- device_train_batch_size:
  32
- device_train_grad_accum:
  1
- device_train_microbatch_size:
  32
- distributed_strategy:
  "fsdp"
- dry_run:
  false
- early_stopping_factor:
  null
- epoch:
  null
- eval_interval:
  1,000
- eval_on_load:
  false
- eval_subset_num_batches:
  -1
- evaluators:[] 0 items
- extra_steps_after_cancel:
  10
- fast_forward_batches:
  null
- force_save_unsharded:
  false
- ▶
  fsdp:{} 5 keys
- fused_loss:
  false
- gen1_gc_interval:
  1
- global_train_batch_size:
  512
- hf_datasets_cache_dir:
  null
- load_path:
  null
- load_path_sharded_checkpointer:
  null
- max_duration:
  262,144
- max_grad_norm:
  1
- max_grad_norm_ratio:
  null
- ▶
  model:{} 43 keys
- module_outputs_save_steps:
  null
- new_style_checkpoints:
  null
- no_pre_train_checkpoint:
  false
- ▶
  optimizer:{} 11 keys
- precision:
  "amp_bf16"
- python_profiling:
  false
- remote_save_folder:
  null
- reset_optimizer_state:
  false
- reset_trainer_state:
  false
- restore_dataloader:
  true
- run_name:
  "1028-rmsnorm-472m"
- save_data_indices:
  true
- save_folder:
  "/network/scratch/m/mirceara/zsl-olmo/out/1028-rmsnorm-472m"
- time_limit:
  null
- ▶
  tokenizer:{} 2 keys
  - identifier:
    "tokenizers/allenai_eleuther-ai-gpt-neox-20b-pii-special.json"
  - truncate_direction:
    "right"
- torch_profiling:
  false
- try_load_latest_save:
  true

Summary metrics are your model's outputs. Learn more

▶
Summary metrics:{} 1052 keys
- optim/clipping_rate:
  0
- optim/exp_avg_sq/transformer.blocks.0.att_proj.weight.avg:
  0.00000000023884794143
- optim/exp_avg_sq/transformer.blocks.0.att_proj.weight.max:
  0.00000143017314258032
- optim/exp_avg_sq/transformer.blocks.0.att_proj.weight.min:
  0.00000000000211669571
- optim/exp_avg_sq/transformer.blocks.0.att_proj.weight.norm:
  0.00001275107206311077
- optim/exp_avg_sq/transformer.blocks.0.attn_out.weight.avg:
  0.00000000060622168396
- optim/exp_avg_sq/transformer.blocks.0.attn_out.weight.max:
  0.00000167225766745105
- optim/exp_avg_sq/transformer.blocks.0.attn_out.weight.min:
  0.00000000001333429027
- optim/exp_avg_sq/transformer.blocks.0.attn_out.weight.norm:
  0.00001206663364428096
- optim/exp_avg_sq/transformer.blocks.0.ff_out.weight.avg:
  0.0000000001437746866
- optim/exp_avg_sq/transformer.blocks.0.ff_out.weight.max:
  0.00000009938728595671
- optim/exp_avg_sq/transformer.blocks.0.ff_out.weight.min:
  0.00000000000067348357
- optim/exp_avg_sq/transformer.blocks.0.ff_out.weight.norm:
  0.00000035226196359872
- optim/exp_avg_sq/transformer.blocks.0.ff_proj.weight.avg:
  0.00000000025963503569
- optim/exp_avg_sq/transformer.blocks.0.ff_proj.weight.max:
  0.000014813716916251
- optim/exp_avg_sq/transformer.blocks.0.ff_proj.weight.min:
  0.00000000000040103871
- optim/exp_avg_sq/transformer.blocks.0.ff_proj.weight.norm:
  0.00003648872734629549
- optim/exp_avg_sq/transformer.blocks.1.att_proj.weight.avg:
  0.00000000045482997924
- optim/exp_avg_sq/transformer.blocks.1.att_proj.weight.max:
  0.00007205313886515796
- optim/exp_avg_sq/transformer.blocks.1.att_proj.weight.min:
  0.00000000000037520933
- optim/exp_avg_sq/transformer.blocks.1.att_proj.weight.norm:
  0.00017947613378055394
- optim/exp_avg_sq/transformer.blocks.1.attn_out.weight.avg:
  0.00000000008024005765
- optim/exp_avg_sq/transformer.blocks.1.attn_out.weight.max:
  0.00000007760765896592
- optim/exp_avg_sq/transformer.blocks.1.attn_out.weight.min:
  0.00000000000592391414
- optim/exp_avg_sq/transformer.blocks.1.attn_out.weight.norm:
  0.00000045021175765214
- optim/exp_avg_sq/transformer.blocks.1.ff_out.weight.avg:
  0.00000000105910558101
- optim/exp_avg_sq/transformer.blocks.1.ff_out.weight.max:
  0.00000605906052442151
- optim/exp_avg_sq/transformer.blocks.1.ff_out.weight.min:
  0.00000000001940792839
- optim/exp_avg_sq/transformer.blocks.1.ff_out.weight.norm:
  0.00004622575579560362
- optim/exp_avg_sq/transformer.blocks.1.ff_proj.weight.avg:
  0.00000000018598234064
- optim/exp_avg_sq/transformer.blocks.1.ff_proj.weight.max:
  0.00000935277421376668
- optim/exp_avg_sq/transformer.blocks.1.ff_proj.weight.min:
  0.00000000000460230595
- optim/exp_avg_sq/transformer.blocks.1.ff_proj.weight.norm:
  0.0000203907438844908
- optim/exp_avg_sq/transformer.blocks.2.att_proj.weight.avg:
  0.0000000003436908258
- optim/exp_avg_sq/transformer.blocks.2.att_proj.weight.max:
  0.00001662587055761833
- optim/exp_avg_sq/transformer.blocks.2.att_proj.weight.min:
  0.00000000000007755347
- optim/exp_avg_sq/transformer.blocks.2.att_proj.weight.norm:
  0.00007962677045725286
- optim/exp_avg_sq/transformer.blocks.2.attn_out.weight.avg:
  0.00000000005725919391
- optim/exp_avg_sq/transformer.blocks.2.attn_out.weight.max:
  0.00000005789939194756
- optim/exp_avg_sq/transformer.blocks.2.attn_out.weight.min:
  0.00000000000753307052
- optim/exp_avg_sq/transformer.blocks.2.attn_out.weight.norm:
  0.00000020600243999525
- optim/exp_avg_sq/transformer.blocks.2.ff_out.weight.avg:
  0.00000000033273184208
- optim/exp_avg_sq/transformer.blocks.2.ff_out.weight.max:
  0.00000152882000747923
- optim/exp_avg_sq/transformer.blocks.2.ff_out.weight.min:
  0.00000000001946807819
- optim/exp_avg_sq/transformer.blocks.2.ff_out.weight.norm:
  0.00001118051386583829
- optim/exp_avg_sq/transformer.blocks.2.ff_proj.weight.avg:
  0.00000000017726416168
- throughput/total_training_Gflops:
  77,906,238,128.02629
- throughput/total_training_log_Gflops:
  25.07877186528228
- train/CrossEntropyLoss:
  2.8206686973571777
- train/Perplexity:
  16.78807305961834

This run produced these artifacts as outputs. Total: 1. Learn more

wandb-history

run-0dvyk2gc-history:v2