Skip to main content

Fluency Scorer

Created on January 24|Last edited on January 27

Definition

Fluency refers to the smoothness and naturalness of the text, focusing on grammatical correctness, appropriate word choice, and the absence of awkward phrasing. It ensures that the text reads well and resembles human language patterns. For instance, a fluent sentence is grammatically correct and easy to understand.

Dataset

We trained a ModerBert-base model on a combination of datasets:
The dataset was annotated with gpt-4o-mini with the following prompt:
You are a helpful assistant that evaluates the fluency of a response.

Fluency refers to the smoothness and naturalness of the text, focusing on grammatical correctness, appropriate word choice, and the absence of awkward phrasing. It ensures that the text reads well and resembles human language patterns. For instance, a fluent sentence is grammatically correct and easy to understand.

Below are examples of fluent and non-fluent sentences.

## Fluent:
- Quantum physics explores subatomic particles.
- The cat is sleeping on the mat.
- The sun is shining brightly.
- She quickly finished her assignment.
- The conference was scheduled for next Monday, and all participants were notified in advance.
- The data analysis revealed significant correlations between the variables,
suggesting a strong relationship that warrants further investigation.

## Non-fluent:
- The cat on the mat sat
- The conference is schedule next Monday, all participant notify advance.
- In summer holiday, children enjoy activity, like swim, hike, visit museum.
Every day bring new adventure, they create memory last lifetime.
- Data analyses reveal significant correlation between variable, suggest strong relation warrant further investigate.
- I look forward weekend; it will relax and fun.

Rate the fluency of the following text, mark it as 1 if it is fluent, 0 if it is not.


Results

If we compare against Claude based LLMFluencyModel we get a good trade-off on performance/speed.

Link to the Compare Dashboard here


Training


Run set
1