llm_surgery

Autometa's group workspace

Group: gemma-7b

1-3

of 3

Tags

Notes

Author

capecape

State

Finished

Start time

February 29th, 2024 4:39:51 PM

Runtime

3h 35m 5s

Tracked hours

3h 34m 28s

Run path

llm_surgery/gemma-zephyr/ouifgy5w

Linux-5.15.0-67-generic-x86_64-with-glibc2.35

Python version

3.11.7

Command

/workspace/eval_harness/run.py --model_path=google/gemma-7b --tasks=hellaswag,arc_challenge,mmlu,winogrande,gsm8k

System Hardware

CPU count	30
Logical CPU count	30
GPU count	1
GPU type	NVIDIA A10

W&B CLI Version

0.16.3

Group

gemma-7b

Job Type

lm_eval

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 2 keys
- ▶
  cli_configs:{} 9 keys
- ▶
  task_configs:{} 61 keys

Summary metrics are your model's outputs. Learn more

▶
Summary metrics:{} 214 keys
- ai2_arc_eval_results:
  "table-file"
- arc_challenge/acc:
  0.49146757679180886
- arc_challenge/acc_norm:
  0.53839590443686
- arc_challenge/acc_norm_stderr:
  0.01456824555029636
- arc_challenge/acc_stderr:
  0.014609263165632186
- arc_challenge/alias:
  "arc_challenge"
- evaluation/eval_results:
  "table-file"
- evaluation/group_eval_results:
  "table-file"
- gsm8k/alias:
  "gsm8k"
- gsm8k/exact_match_stderr,flexible-extract:
  0.013741096412226754
- gsm8k/exact_match_stderr,strict-match:
  0.013748996794921798
- gsm8k/exact_match,flexible-extract:
  0.533737680060652
- gsm8k/exact_match,strict-match:
  0.5291887793783169
- hellaswag/acc:
  0.6047600079665405
- hellaswag/acc_norm:
  0.806512646883091
- hellaswag/acc_norm_stderr:
  0.003942243077246894
- hellaswag/acc_stderr:
  0.004879030010598915
- hellaswag/alias:
  "hellaswag"
- math_word_problems_eval_results:
  "table-file"
- mmlu_abstract_algebra/acc:
  0.29
- mmlu_abstract_algebra/acc_stderr:
  0.04560480215720684
- mmlu_abstract_algebra/alias:
  " - abstract_algebra"
- mmlu_anatomy/acc:
  0.5851851851851851
- mmlu_anatomy/acc_stderr:
  0.042561937679014075
- mmlu_anatomy/alias:
  " - anatomy"
- mmlu_astronomy/acc:
  0.7302631578947368
- mmlu_astronomy/acc_stderr:
  0.03611780560284898
- mmlu_astronomy/alias:
  " - astronomy"
- mmlu_business_ethics/acc:
  0.63
- mmlu_business_ethics/acc_stderr:
  0.048523658709390974
- mmlu_business_ethics/alias:
  " - business_ethics"
- mmlu_clinical_knowledge/acc:
  0.6754716981132075
- mmlu_clinical_knowledge/acc_stderr:
  0.02881561571343211
- mmlu_clinical_knowledge/alias:
  " - clinical_knowledge"
- mmlu_college_biology/acc:
  0.75
- mmlu_college_biology/acc_stderr:
  0.03621034121889507
- mmlu_college_biology/alias:
  " - college_biology"
- mmlu_college_chemistry/acc:
  0.46
- mmlu_college_chemistry/acc_stderr:
  0.05009082659620332
- mmlu_college_chemistry/alias:
  " - college_chemistry"
- mmlu_college_computer_science/acc:
  0.52
- mmlu_college_computer_science/acc_stderr:
  0.050211673156867795
- mmlu_college_computer_science/alias:
  " - college_computer_science"
- mmlu_college_mathematics/acc:
  0.39
- mmlu_college_mathematics/acc_stderr:
  0.04902071300001974
- mmlu_college_mathematics/alias:
  " - college_mathematics"
- winogrande_eval_results:
  "table-file"
- winogrande/acc:
  0.7387529597474349
- winogrande/acc_stderr:
  0.012346914863415308
- winogrande/alias:
  "winogrande"