Transformer - Greedy vs Beam Search
INF8225 - TP3 - Greedy and Beam search with the Transformer (2 tests)
by Renaud Lespérance and Morgan Péju
Created on April 8|Last edited on April 8
Comment
Tests description and results
We're using the bests parameters determined thanks to the previous tests:
- batch_size = 128
- dim_embedding = 300
- positional_embedding : classic (nn.Embedding)
- activation function = "ELU"
- Scale = 6, Layers = 2
- We did those tests on 10 epochs
We wanted to test two methods for producing a sequence of tokens from an autoregressive model :
- Greedy search : choose the most probable token as the unique and only next target token, and repeat this processus until the <eos> token is predicted.
- Beam search : selects the k-most probable tokens at each step. From those k tokens, the current sequence is duplicated k times and the k tokens are appended to the k sequences to produce new k sequences.
To assess each method, ce decided to implement the BLUE SCORE. Blue score is a metric for evaluating a generated sentence to a reference sentence. A perfect match results in a score of 1.0. It works by counting matching n-grams in the predicted sentence to n-grams in the reference text, where 1-gram would be each token and a bigram comparison would be each word pair, etc. The comparison is made regardless of word order.
As we can see in the graphs below and as we expected, the Beam Search offers better results in term of pure text translation. Indeed, its blue scores are always higher than greedy search's. Moreover, we can see that our model is learning: the blue scores are increasing through the epochs.
Run set
2
Add a comment