BLEU Scorer

Definition

Precision: The fraction of n-grams in the hypothesis that appear in the references. By default we use 1-4 grams.
Brevity Penalty: A penalty to prevent short hypotheses from scoring disproportionately high.

Evaluates the BLEU score for a single prediction against its references.
Useful for debugging individual predictions or analyzing specific outputs.
The average of sentence-level BLEU is not reliable because averaging these individual scores corresponds to macro-averaging, as it treats each sentence equally regardless of length.

Aggregates statistics across all hypotheses and references before calculating BLEU. This method reflects micro-averaging, as it accounts for the contribution of each sentence proportionally to its length.
Provides a holistic view of model performance across the entire dataset.
The original BLEU metric, as introduced by Papineni et al. (2002), employs corpus-level (micro-average) precision.

Usage

from weave import Evaluations
from weave.scorers import BLEUScorer

bleu_scorer = BLEUScorer()

evaluation = Evaluation(
    dataset=dataset,
    scorers=[BLEUScorer(),]
)

# ...run evaluation

We return sentence-level scores so that you can look at the prediction quality at a fine-grained level. As a summary we also return the average of the sentence-level BLEU scores.

Figure 1: Sample wise view of the evaluation on the TruthfulQA (generation) benchmark where our scorer gives sentence-level BLEU scores. >>>Click here for interactivity<<<

We return corpus-level score as well. This is the reliable (original implementation) BLEU score as reported in most research papers. Because this is a score given to the entire dataset, you will find the score in the summary of the evaluation.

Figure 2: The BLEU score. >>>Click here for interactivity<<<

We provide you with knobs so that you can configure the scorer. The defaults are provided by the scareBLEU library and in most cases need not be changed. If you know what you are doing, you have the options to configure the scorer.

Limitations

String Matching Focus: BLEU evaluates string similarity, not true translation quality or semantic equivalence.
Single Reference Limitation: Many tests use only one human reference translation, penalizing valid translations that differ in phrasing or word choice.
Ignores Synonyms and Paraphrases: BLEU gives no credit for synonyms or paraphrases (e.g., “sofa” vs. “couch”), leading to lower scores for accurate translations.
Syntactically Incorrect Matches: Nonsensical outputs with correct n-grams in the wrong order can score high.
Meaningless for Cross-Domain/Language Comparisons: Scores are highly specific to the test set and language pair, making cross-domain or cross-language comparisons invalid.
Insensitive to Meaning: BLEU does not assess if the translation preserves meaning; it only measures token overlap.
Unreliable Small Differences: Small score differences are hard to interpret and often not meaningful.
Overfitting Risk: Systems trained on the test set can artificially inflate BLEU scores without real-world improvements.
Limited by Reference Set: Using multiple human references improves scores but still cannot fully capture translation diversity.

Closing remarks