llm-optimizer-benchmark Workspace – Weights & Biases

Skip to main content

Semenov-andrei-v's workspace

Charts

12

64x1_model-moe_llama_dataset-fineweb_opt-prodigy__iterations-336000_warmup_steps-2000_lr-1.0_beta2-0.999_grad_clip-0.5_prodigy_use_bias_correction-True_n_layer-12_moe-True

64x1_model-moe_llama_dataset-fineweb_opt-prodigy__iterations-42000_warmup_steps-2000_lr-1.0_beta2-0.999_grad_clip-0.5_prodigy_use_bias_correction-True_n_layer-12_moe-True

64x1_model-moe_llama_dataset-fineweb_opt-prodigy__iterations-336000_warmup_steps-2000_lr-1.0_beta2-0.999_grad_clip-0.5_prodigy_use_bias_correction-True_n_layer-12_moe-True

64x4(rank=0)_model-llama_dataset-fineweb_opt-prodigy__iterations-180000_warmup_steps-2000_lr-1.0_beta2-0.9999_grad_clip-0.5_prodigy_use_bias_correction-True

64x1(rank=0)_model-llama_dataset-fineweb_opt-prodigy__iterations-180000_warmup_steps-2000_lr-1.0_beta2-0.9999_grad_clip-0.5_prodigy_use_bias_correction-True

32x1(rank=0)_model-llama_dataset-fineweb_opt-prodigy__iterations-256000_lr-1.0_beta2-0.999_grad_clip-0.5_prodigy_use_bias_correction-True

62x8_model-llama_dataset-fineweb_opt-prodigy__iterations-8000_warmup_steps-2000_lr-5.0_beta2-0.999_grad_clip-0.1_prodigy_use_bias_correction-True_n_head-16_n_layer-12_n_embd-2048

62x8_model-llama_dataset-fineweb_opt-prodigy__log_parameter_norms-True_iterations-8000_warmup_steps-2000_lr-10.0_beta2-0.999_grad_clip-0.1_prodigy_use_bias_correction-True_n_head-16_n_layer-12_n_embd-2048

62x8_model-llama_dataset-fineweb_opt-prodigy__iterations-16000_warmup_steps-2000_lr-1.0_beta1-0.95_beta2-0.99_grad_clip-0.1_prodigy_use_bias_correction-True_n_head-16_n_layer-12_n_embd-2048

62x8_model-llama_dataset-fineweb_opt-prodigy__iterations-16000_warmup_steps-2000_lr-5.0_beta1-0.95_beta2-0.99_grad_clip-0.1_prodigy_use_bias_correction-True_n_head-16_n_layer-12_n_embd-2048

Selected runs are not logging media for the key final-val/perplexity, but instead are logging values of type nil.

If final-val/perplexity is never supposed to be a media type, please delete this panel and create the proper panel type manually.

Selected runs are not logging media for the key final-val/loss, but instead are logging values of type nil.

If final-val/loss is never supposed to be a media type, please delete this panel and create the proper panel type manually.

Selected runs are not logging media for the key final-val/acc, but instead are logging values of type nil.

If final-val/acc is never supposed to be a media type, please delete this panel and create the proper panel type manually.

Showing first 10 runs

62x8_model-moe_llama_dataset-fineweb_opt-soap__iterations-10000_warmup_steps-2000_beta2-0.999_grad_clip-0.1_moe-True

64x1_model-moe_llama_dataset-fineweb_opt-adopt__iterations-336000_warmup_steps-2000_beta2-0.999_grad_clip-0.5_n_layer-12_moe-True

62x8_model-moe_llama_dataset-fineweb_opt-adamw__iterations-10000_warmup_steps-2000_beta2-0.999_grad_clip-0.1_moe-True

64x1_model-moe_llama_dataset-fineweb_opt-signum__iterations-336000_warmup_steps-2000_lr-0.0005_grad_clip-0.5_momentum-0.95_nesterov-True_mars_vr_gamma-0.024_n_layer-12_moe-True

64x1_model-moe_llama_dataset-fineweb_opt-d-muon__iterations-336000_warmup_steps-2000_beta1-0.8_beta2-0.999_grad_clip-0.5_momentum-0.95_nesterov-True_mars_vr_gamma-0.024_n_layer-12_moe-True

32x2_model-moe_llama_dataset-fineweb_opt-sophiag__iterations-336000_warmup_steps-2000_lr-0.0005_beta2-0.999_grad_clip-0.5_mars_vr_gamma-0.024_n_layer-12_moe-True

64x1_model-moe_llama_dataset-fineweb_opt-soap__iterations-336000_warmup_steps-2000_beta2-0.999_grad_clip-0.5_mars_vr_gamma-0.024_n_layer-12_moe-True

64x1_model-moe_llama_dataset-fineweb_opt-soap__iterations-336000_warmup_steps-2000_beta2-0.999_grad_clip-0.5_n_layer-12_moe-True

64x1_model-moe_llama_dataset-fineweb_opt-d-muon__iterations-336000_warmup_steps-2000_beta1-0.8_beta2-0.999_grad_clip-0.5_momentum-0.95_nesterov-True_n_layer-12_moe-True

32x2_model-moe_llama_dataset-fineweb_opt-sophiag__iterations-336000_warmup_steps-2000_lr-0.0005_beta2-0.999_grad_clip-0.5_n_layer-12_moe-True