Who Is Them? Text Disambiguation With Transformers

Table of Contents

Overview

Human Language Is Ambiguous to Machines

A: "Humans train deep nets to help them in the real world."
B: "Humans train deep nets to use them in the real world."

Language Understanding Benchmarks for Measuring Progress

HuggingFace Transformers are a great place to start

First pass: Fine-tuning BERT baseline

Short manual hyperparameter exploration

Helpful Modifications

increase logging frequency: this doesn't affect the model performance, but it yields more detailed/less noisy plots (compare the red zigzag with the peach curve).
lower the intial learning rate: my initial runs didn't seem to learn much; instead, eval_acc dropped substantially over training. Lowering the leaning rate helped stabilize the accuracy and even increase it slightly
lower maximum sequence length: The starting window size of 128 seemed too large for the space of Winograd Schema sentences (perhaps optimized for other tasks in the benchmarks). Lowering to 64, 32, and finally 16 kept improving eval_acc. At 8, I saw eval_acc drop again.
more epochs: increasing the number of epochs to 10 and then 20 gave me a better sense for model performance without wasting too much time on extra epochs after eval_acc stopped improving. Note that the combination of training batch size (per_gpu_train_batch_size) and number of epochs (num_train_epochs) affect how many training steps we effectively see, hence the endpoints of the curves above are different)

Example Script to Run Training

export GLUE_DIR=/path/to/glue/data
export TASK_NAME=WNLI

python run_glue.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --task_name $TASK_NAME \
  --do_train \
  --do_eval \
  --evaluate_during_training \
  --data_dir $GLUE_DIR/$TASK_NAME \
  --max_seq_length 128 \
  --per_gpu_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir /tmp/$TASK_NAME/ \
  --overwrite_output_dir \
  --logging_steps 10


Second Pass: Comparing Different Base Models

How Useful Is It To Fine-Tune Yet Another BERT Model?

Roberta, a more robustly optimized variant of BERT, performs well on WNLI according to a model card in the Transformers repo, especially with pretraining on extra data
Albert, a lighter (parameter-reduced) version of BERT with a self-supervised loss that helps with inter-sentence coherence and especially tasks with multi-sentence inputs. This is the base model for the top entry in the regular GLUE leaderboard, and it ties several models for best performance on WNLI at 94.5 (surprisingly better than the T5 score from SuperGLUE). You can read more in Lan et al 2019 from Google Research. Interestingly, in the source repo, WNLI is left out of the benchmarked performance on GLUE, and the improvement on MNLI is fairly small.
XLNet, an improvement on BERT that trains to predict all words (instead of just the 15% of masked words) in random order instead of sequential order. This might give XLNet an advantage in understanding dependencies between different words in a sentence. Though XLNet generally outperforms BERT on natural language inference tasks, results for WNLI are not reported in the source repository.

Observations

overfitting happens very quickly: across these experiments, eval_acc is often noisy and defaults to _decreasing_ over training. Setting the learning rate lower helps, except the default learning rate schedule for Transformers decreases this already low learning rate further over time, making it tricky to continue improving. The evaluation loss often diverges (see chart above), but it's possible to find a combination of hyperparameters that causes the evaluation loss to converge slowly instead.
XLNet seems to learn the most: training loss doesn't decrease very much in any of these runs, but the XLNet models on average have the most canonical training curves _and_ evaluation loss curves, paired with reasonable accuracies
Roberta performs reasonably: easiest of the models to tune manually, though overfits quickly if you look at the evaluation loss curves
Albert is too complicated: this base requires more degrees of freedom or a hyperparameter sweep, not quick exploration by hand. Results seemed the least consistent here, especially because the later/supposedly better base model v2 seems to perform worse on this task than v1.

Transformers TL;DR: Tune Less, They’re Already Powerful

Albert-v1 Is Best on Average, XLNet Has Highest Accuracy.

Next steps: T5, Ernie

T5, the Text-to-Text Transfer Transformer from Raffel et al at Google, is the state-of-the-art in transfer learning for NLP and scores the highest overall on the SuperGLUE leaderboard, an improvement on the GLUE leaderboard referenced earlier.
Ernie, the Enhanced Representation through Knowledge Integration model from Sun et al 2019, which enriches its language learning with entity-level and phrase-level masking for concepts composed of multiple words. This improves performance on natural language inference tasks and ties for first on Winograd NLI in the GLUE leaderboard.

Other Base Models to Explore in Transformers

DistilBERT
Camembert
XLMRoberta
Bart
FlaubertConfig
XLM

Concrete Examples from Best Model

Each concept has many possible Winograd schemas with binary output labels

"Humans train deep nets to help them in the real world. Deep nets help humans." Label: 1 (True, the second sentence follows from, or is entailed by, the first)
"Humans train deep nets to use them in the real world. Deep nets use humans." Label: 0 (False, the second sentence doesn't follow)

Observations

the data is a bit noisy/some labels are debatable: Some of the examples may be mislabeled in the dev set, or perhaps more ambiguous to humans than an author thought. "They broadcast an announcement, but a subway came into the station and I couldn't hear over it. I couldn't hear the subway." and "Fred is the only man still alive who remembers my great-grandfather. He is a remarkable man. Fred is a remarkable man." are both marked as "True, this follows", but I don't think this is super clear in either case.
weak understanding of animal/entity relationships: Some of the harder examples involve two animals, like a duck and a minnow, or a cat and a mouse, where a human would know the correct answer based on how these animals relate in the real world (cats hunt mice, ducks probably eat minnows--perhaps this is an issue for the "police" and "gang members" example too). Explicitly increasing the knowledge representation of the model for certain entity types may help here, as seems to be the approach with Ernie.
syntax may be easier to learn than semantics: Most of the highest-confidence correct answers have a repeating pattern, where the subject is the first word in each sentence, the final clause is slightly longer and phrases identically in both sentences, and the label is "0": for example, "Sam took French classes from Adam, because he was known to speak it fluently. Sam was known to speak it fluently." When the first word in the second sentence is different and is not the subject, such as with the two examples involving Beth and Sally in the first table, the model seems to get the answer wrong.
proper nouns are a confound: "Madonna" is a fairly rare word, and two of the highest confidence correct answers involve Madonna. Several formulations of the Madonna example appear in the train set. Perhaps the model gets more Madonna examples correct because it overfits to that rare token, or perhaps to the baseline rate of the labels for sentences that include Madonna. However, in train, Madonna appears in 2 examples with label 0 and 4 with label 1, which doesn't support the overfitting hypothesis. This also brings up a broader question of which facts about the world we expect a model to know or care about it knowing: the names of specific celebrities? whether particular names refer to celebrities? just the concept of celebrities?
next steps: more rigorous data analysis: It would be useful to build a frequency histogram on all the proper names and other subjects that appear in this WNLI, as well as the different possible formulations of an example, and balance across all of these. This would help disentangle whether a model is learning accidental patterns in the construction of the examples, or actual meaning.

Model exploration summary

Observations

lower learning rates are better: these models are already well-trained and they easily overfit the small dataset (note that "start_lr" and "learning_rate" are two column names for the same hyperparameter—"learning rate" is also a default variable that changes over time as a Transformer trains)
shorter sequence length is better: would need to dig into concrete examples, but they tend to be short
larger batch size is better: I did not focus on this hyperparameter, but increasing it seems to help performance—though this too is inconclusive, as larger batch size will lead to fewer log steps and less overfitting
more epochs are slightly better: again because the models ovefit easily, this is inconclusive