amr-amr

Amr-amr's group workspace

Group: 285M

1-8

of 8

Tags

Main Results (OLMo)

Author

amr-amr

State

Finished

Start time

November 2nd, 2024 8:36:11 PM

Runtime

9h 23m 9s

Tracked hours

9h 23m 8s

Run path

amr-amr/zsl/ng3zi0sb

Linux-5.15.0-101-generic-x86_64-with-glibc2.35

Python version

3.10.14

Git repository

git clone git@github.com:mirandrom/zsl-olmo.git

Git state

git checkout -b "1028-rmsnorm-285m" a5d85f50b6398187a0f97353c8dff998342638b3

Command

/home/mila/m/mirceara/proj/zsl-olmo/scripts/train.py /home/mila/m/mirceara/proj/zsl-olmo/zsl/exp/1028-rmsnorm/285m.yaml --run_name=1028-rmsnorm-285m --wandb.name=1028-rmsnorm-285m --wandb.group=1028-rmsnorm-285m --wandb.project=zsl-test --wandb.entity=amr-amr

System Hardware

CPU count	48
Logical CPU count	48
GPU count	2
GPU type	[NVIDIA L40S, NVIDIA L40S]

W&B CLI Version

0.18.1

Group

285M

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 64 keys
- activation_checkpointing:
  "fine_grained"
- auxiliary_loss_multiplier:
  0.0001
- canceled_check_interval:
  50
- ▶
  compile:{} 3 keys
  - backend:
    "inductor"
  - fullgraph:
    false
  - mode:
    "default"
- console_log_interval:
  1
- ▶
  data:{} 15 keys
- ddp:
  null
- device_eval_batch_size:
  16
- device_train_batch_size:
  32
- device_train_grad_accum:
  1
- device_train_microbatch_size:
  32
- distributed_strategy:
  "fsdp"
- dry_run:
  false
- early_stopping_factor:
  null
- epoch:
  null
- eval_interval:
  1,000
- eval_on_load:
  false
- eval_subset_num_batches:
  -1
- evaluators:[] 0 items
- extra_steps_after_cancel:
  10
- fast_forward_batches:
  null
- force_save_unsharded:
  false
- ▶
  fsdp:{} 5 keys
- fused_loss:
  false
- gen1_gc_interval:
  1
- global_train_batch_size:
  512
- hf_datasets_cache_dir:
  null
- load_path:
  null
- load_path_sharded_checkpointer:
  null
- max_duration:
  262,144
- max_grad_norm:
  1
- max_grad_norm_ratio:
  null
- ▶
  model:{} 43 keys
- module_outputs_save_steps:
  null
- new_style_checkpoints:
  null
- no_pre_train_checkpoint:
  false
- ▶
  optimizer:{} 11 keys
- precision:
  "amp_bf16"
- python_profiling:
  false
- remote_save_folder:
  null
- reset_optimizer_state:
  false
- reset_trainer_state:
  false
- restore_dataloader:
  true
- run_name:
  "1028-rmsnorm-285m"
- save_data_indices:
  true
- save_folder:
  "/network/scratch/m/mirceara/zsl-olmo/out/1028-rmsnorm-285m"
- time_limit:
  null
- ▶
  tokenizer:{} 2 keys
  - identifier:
    "tokenizers/allenai_eleuther-ai-gpt-neox-20b-pii-special.json"
  - truncate_direction:
    "right"
- torch_profiling:
  false
- try_load_latest_save:
  true

Summary metrics are your model's outputs. Learn more

▶
Summary metrics:{} 1052 keys
- optim/clipping_rate:
  0
- optim/exp_avg_sq/transformer.blocks.0.att_proj.weight.avg:
  0.00000000124075749675
- optim/exp_avg_sq/transformer.blocks.0.att_proj.weight.max:
  0.00000797734355728608
- optim/exp_avg_sq/transformer.blocks.0.att_proj.weight.min:
  0.00000000000327517354
- optim/exp_avg_sq/transformer.blocks.0.att_proj.weight.norm:
  0.00003849484710372053
- optim/exp_avg_sq/transformer.blocks.0.attn_out.weight.avg:
  0.00000000248471376807
- optim/exp_avg_sq/transformer.blocks.0.attn_out.weight.max:
  0.00000684986480337102
- optim/exp_avg_sq/transformer.blocks.0.attn_out.weight.min:
  0.00000000002091148088
- optim/exp_avg_sq/transformer.blocks.0.attn_out.weight.norm:
  0.00004172179978922941
- optim/exp_avg_sq/transformer.blocks.0.ff_out.weight.avg:
  0.00000000054179766229
- optim/exp_avg_sq/transformer.blocks.0.ff_out.weight.max:
  0.00000404886623073253
- optim/exp_avg_sq/transformer.blocks.0.ff_out.weight.min:
  0.00000000000384256906
- optim/exp_avg_sq/transformer.blocks.0.ff_out.weight.norm:
  0.00000768183781474363
- optim/exp_avg_sq/transformer.blocks.0.ff_proj.weight.avg:
  0.00000000104010233759
- optim/exp_avg_sq/transformer.blocks.0.ff_proj.weight.max:
  0.00003091021062573418
- optim/exp_avg_sq/transformer.blocks.0.ff_proj.weight.min:
  0.00000000000025768192
- optim/exp_avg_sq/transformer.blocks.0.ff_proj.weight.norm:
  0.0000691653840476647
- optim/exp_avg_sq/transformer.blocks.1.att_proj.weight.avg:
  0.00000000097136720889
- optim/exp_avg_sq/transformer.blocks.1.att_proj.weight.max:
  0.00005671779945259914
- optim/exp_avg_sq/transformer.blocks.1.att_proj.weight.min:
  0.00000000000064649306
- optim/exp_avg_sq/transformer.blocks.1.att_proj.weight.norm:
  0.0001834071590565145
- optim/exp_avg_sq/transformer.blocks.1.attn_out.weight.avg:
  0.00000000019065145296
- optim/exp_avg_sq/transformer.blocks.1.attn_out.weight.max:
  0.00000033353191497554
- optim/exp_avg_sq/transformer.blocks.1.attn_out.weight.min:
  0.00000000001444518116
- optim/exp_avg_sq/transformer.blocks.1.attn_out.weight.norm:
  0.00000127944497307908
- optim/exp_avg_sq/transformer.blocks.1.ff_out.weight.avg:
  0.00000000018286858139
- optim/exp_avg_sq/transformer.blocks.1.ff_out.weight.max:
  0.00000062301023717737
- optim/exp_avg_sq/transformer.blocks.1.ff_out.weight.min:
  0.00000000001192923017
- optim/exp_avg_sq/transformer.blocks.1.ff_out.weight.norm:
  0.00000095932284693845
- optim/exp_avg_sq/transformer.blocks.1.ff_proj.weight.avg:
  0.00000000057512383744
- optim/exp_avg_sq/transformer.blocks.1.ff_proj.weight.max:
  0.00001250297373189824
- optim/exp_avg_sq/transformer.blocks.1.ff_proj.weight.min:
  0.00000000000148823586
- optim/exp_avg_sq/transformer.blocks.1.ff_proj.weight.norm:
  0.00003772940544877201
- optim/exp_avg_sq/transformer.blocks.2.att_proj.weight.avg:
  0.00000000100016050997
- optim/exp_avg_sq/transformer.blocks.2.att_proj.weight.max:
  0.00010575064516160636
- optim/exp_avg_sq/transformer.blocks.2.att_proj.weight.min:
  0.0000000000011034692
- optim/exp_avg_sq/transformer.blocks.2.att_proj.weight.norm:
  0.0002198048168793321
- optim/exp_avg_sq/transformer.blocks.2.attn_out.weight.avg:
  0.00000000012454764997
- optim/exp_avg_sq/transformer.blocks.2.attn_out.weight.max:
  0.00000060745816199415
- optim/exp_avg_sq/transformer.blocks.2.attn_out.weight.min:
  0.0000000000167108688
- optim/exp_avg_sq/transformer.blocks.2.attn_out.weight.norm:
  0.00000123838196941506
- optim/exp_avg_sq/transformer.blocks.2.ff_out.weight.avg:
  0.00000000112060360991
- optim/exp_avg_sq/transformer.blocks.2.ff_out.weight.max:
  0.00000451339747087332
- optim/exp_avg_sq/transformer.blocks.2.ff_out.weight.min:
  0.00000000002819950523
- optim/exp_avg_sq/transformer.blocks.2.ff_out.weight.norm:
  0.00003103411290794611
- optim/exp_avg_sq/transformer.blocks.2.ff_proj.weight.avg:
  0.00000000036945391191
- throughput/total_training_Gflops:
  54,862,691,626.57835
- throughput/total_training_log_Gflops:
  24.7280993848064
- train/CrossEntropyLoss:
  2.908607482910156
- train/Perplexity:
  18.331254200909473

This run consumed these artifacts as inputs. Learn more

No rows found

This run produced these artifacts as outputs. Total: 1. Learn more

wandb-history

run-ng3zi0sb-history:v0