Eisvogel: Evaluating German Language Proficiency

A kingfisher (Alcedo atthis) with a wooden plaque.

🇩🇪 German LLM Leaderboard (01/01/2025)

Glossary

Mean Win Rate (MWR): The mean win rate is a metric that reflects how often a model achieves a higher score than another model across various scenarios. For each scenario, we calculate the default accuracy metric—like exact match or F1 score—and then, instead of averaging these different metrics (which may have distinct scales and interpretations), we determine how frequently the model outperforms another. The mean win rate is the average of these performance comparisons across all scenarios, making it a meaningful measure even when different metrics vary in scale or units. However, it’s important to note that mean win rate only makes sense in the context of a set of models being compared and doesn’t carry meaning when interpreted in isolation.
EM: The exact match metric evaluates correctness by checking if the model’s output matches the reference answer precisely, character by character. For an exact match, the generated output must be identical to the reference answer as a string, with no variations or discrepancies allowed.

Technical Details/Methodology

Multilingual Massive Multitask Language Understanding (MMMLU)
Multilingual Grade School Math Benchmark (MGSM)

Multilingual Massive Multitask Language Understanding (MMMLU)

Technical details

The MMMLU benchmark is created by OpenAI by translating the original MMLU benchmark using human annotators. We selected this benchmark because of the higher degree of translation accuracy. The complete MMMLU benchmark contains 14 languages. Check out the data card here. Ofcourse we are using only the DE_DE subset of the dataset for this leaderboard.
The translation is done only for the test set. Since we do not have access to translated train or validation set, we are evaluating in zero-shot setting across all models.
The system prompt is given below:

Beantworten Sie die folgenden Multiple-Choice-Fragen zu {}. Jede Frage hat vier Antwortmöglichkeiten: A, B, C oder D. Wählen Sie die passendste Antwort und geben Sie nur den entsprechenden Buchstaben an.

This system prompt is a modification [explain modification in appendix] from MMLU where we are explicitly asking the model to return the response as A, B, C or D. Not doing so in zero shot setting was leading to incomparable results due to varying verbosity of the models. We want to acknowledge that a singular system prompt for every model is not the best way to check the encoded knowledge but explicitly "asking to do what is intended gives a fair chance to each model".
Exact Match (EM) is used as the choice of metric. The correctness condition for exact match is that the model generation match the correct reference exactly as strings.

Results

{
    "STEM": ['abstract_algebra', 'college_biology', 'college_chemistry', 'college_physics', 'astronomy', 'high_school_biology', 'high_school_chemistry', 'high_school_physics', 'high_school_mathematics', 'machine_learning', 'formal_logic', 'college_computer_science', 'high_school_computer_science', 'computer_security'],
    "Medical & Health": ['clinical_knowledge', 'college_medicine', 'medical_genetics', 'virology', 'human_aging', 'nutrition', 'professional_medicine', 'professional_psychology', 'human_sexuality'],
    "Social Sciences": ['high_school_us_history', 'high_school_world_history', 'sociology', 'public_relations', 'philosophy', 'jurisprudence', 'international_law', 'high_school_government_and_politics'],
    "Business & Economics": ['business_ethics', 'econometrics', 'management', 'marketing', 'high_school_macroeconomics', 'high_school_microeconomics', 'professional_accounting'],
    "Law & Ethics": ['moral_disputes', 'moral_scenarios', 'international_law', 'jurisprudence', 'philosophy', 'legal_ethics', 'security_studies']
}

MGSM

Technical Details

We are using 5-shot examples setting to evaluate the LLMs. (We could have used up to 8-shot examples)
We are evaluating using both Chain of Thought (CoT) reasoning and without it.
The default max_tokens used for non CoT evaluation is 400. For CoT we are using max_tokens=600.
The prompt structure for non CoT evaluation is:

Frage: <few shot example question 1>
A: Die Antwort ist <ans 1>

Frage: <few shot example question 2>
A: Die Antwort ist <ans 2>

Frage: <few shot example question 3>
A: Die Antwort ist <ans 3>

Frage: <few shot example question 4>
A: Die Antwort ist <ans 4>

Frage: <few shot example question <5>
A: Die Antwort is <ans 5>

Frage: <question>
A:

The prompt structure for CoT based evaluation is:

Frage: <few shot example question 1>
A: <ans 1 with reasoning step>

Frage: <few shot example question 2>
A: <ans 2 with reasoning step>

Frage: <few shot example question 3>
A: <ans 3 with reasoning step>

Frage: <few shot example question 4>
A: <ans 4 with reasoning step>

Frage: <few shot example question <5>
A: <ans 5 with reasoning step>

Frage: <question>
A:

Results

Efficiency

If you are looking for blazing fast model, Gemini 1.5 Flash variants are really good. However, they aren't very performant on the benchmarks as shown in the next section.
The Claude variant of models are quite slow. They are performant but given few models are equally performant (even better) and faster, the Claude looses its charm.
The Mistral and GPT model variants hits that sweet spot.

Accuracy vs Efficiency (Mean Win Rate)

On the left side of this scatter plot, the Gemini Flash series of models are fast but not that accurate on our benchmark.
On the right side of this scatter plot, the Claude 3.5 sonnet is the slowest model but comes with decent performance on the benchmark.
Mistral Large models are performing really well on our benchmark, with Mistral Large (2411) variant hitting that sweet spot of high accuracy and decent latency. GPT-4o closely follows this sweet spot.

Conclusion