llm_surgery

Autometa's group workspace

Group: gemma-2b

1-3

of 3

Tags

Notes

Author

capecape

State

Finished

Job

eval_harness_wandb:v1

Start time

February 29th, 2024 8:15:19 PM

Runtime

1h 34m 33s

Tracked hours

1h 33m 11s

Run path

llm_surgery/gemma-zephyr/0i7fb9dk

Linux-5.15.0-67-generic-x86_64-with-glibc2.35

Python version

3.11.7

Command

/workspace/eval_harness/run.py --model_path=google/gemma-2b --tasks=hellaswag,arc_challenge,mmlu,winogrande,gsm8k

System Hardware

CPU count	30
Logical CPU count	30
GPU count	1
GPU type	NVIDIA A10

W&B CLI Version

0.16.3

Group

gemma-2b

Job Type

lm_eval

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 2 keys
- ▶
  cli_configs:{} 9 keys
- ▶
  task_configs:{} 61 keys

Summary metrics are your model's outputs. Learn more

▶
Summary metrics:{} 214 keys
- ai2_arc_eval_results:
  "table-file"
- arc_challenge/acc:
  0.4035836177474403
- arc_challenge/acc_norm:
  0.4215017064846416
- arc_challenge/acc_norm_stderr:
  0.014430197069326026
- arc_challenge/acc_stderr:
  0.014337158914268448
- arc_challenge/alias:
  "arc_challenge"
- evaluation/eval_results:
  "table-file"
- evaluation/group_eval_results:
  "table-file"
- gsm8k/alias:
  "gsm8k"
- gsm8k/exact_match_stderr,flexible-extract:
  0.010609827611527362
- gsm8k/exact_match_stderr,strict-match:
  0.010609827611527362
- gsm8k/exact_match,flexible-extract:
  0.1811978771796816
- gsm8k/exact_match,strict-match:
  0.1811978771796816
- hellaswag/acc:
  0.5277833100975902
- hellaswag/acc_norm:
  0.7143995220075682
- hellaswag/acc_norm_stderr:
  0.004507768029590029
- hellaswag/acc_stderr:
  0.0049820721084480836
- hellaswag/alias:
  "hellaswag"
- math_word_problems_eval_results:
  "table-file"
- mmlu_abstract_algebra/acc:
  0.19
- mmlu_abstract_algebra/acc_stderr:
  0.039427724440366206
- mmlu_abstract_algebra/alias:
  " - abstract_algebra"
- mmlu_anatomy/acc:
  0.34074074074074073
- mmlu_anatomy/acc_stderr:
  0.04094376269996793
- mmlu_anatomy/alias:
  " - anatomy"
- mmlu_astronomy/acc:
  0.32894736842105265
- mmlu_astronomy/acc_stderr:
  0.03823428969926605
- mmlu_astronomy/alias:
  " - astronomy"
- mmlu_business_ethics/acc:
  0.38
- mmlu_business_ethics/acc_stderr:
  0.04878317312145632
- mmlu_business_ethics/alias:
  " - business_ethics"
- mmlu_clinical_knowledge/acc:
  0.3433962264150943
- mmlu_clinical_knowledge/acc_stderr:
  0.02922452646912479
- mmlu_clinical_knowledge/alias:
  " - clinical_knowledge"
- mmlu_college_biology/acc:
  0.3680555555555556
- mmlu_college_biology/acc_stderr:
  0.04032999053960719
- mmlu_college_biology/alias:
  " - college_biology"
- mmlu_college_chemistry/acc:
  0.35
- mmlu_college_chemistry/acc_stderr:
  0.0479372485441102
- mmlu_college_chemistry/alias:
  " - college_chemistry"
- mmlu_college_computer_science/acc:
  0.3
- mmlu_college_computer_science/acc_stderr:
  0.046056618647183814
- mmlu_college_computer_science/alias:
  " - college_computer_science"
- mmlu_college_mathematics/acc:
  0.33
- mmlu_college_mathematics/acc_stderr:
  0.04725815626252604
- mmlu_college_mathematics/alias:
  " - college_mathematics"
- winogrande_eval_results:
  "table-file"
- winogrande/acc:
  0.6440410418310971
- winogrande/acc_stderr:
  0.013456740656273955
- winogrande/alias:
  "winogrande"