neftune for summarization - longt5

about

We show that language model finetuning can be improved, sometimes dramatically, with a simple augmentation. NEFTune adds noise to the embedding vectors during training. Standard finetuning of LLaMA-2-7B using Alpaca achieves 29.79% on AlpacaEval, which rises to 64.69% using noisy embeddings. NEFTune also improves over strong baselines on modern instruction datasets. Models trained with Evol-Instruct see a 10% improvement, with ShareGPT an 8% improvement, and with OpenPlatypus an 8% improvement. Even powerful models further refined with RLHF such as LLaMA-2-Chat benefit from additional training with NEFTune.

what

there is a new technique/method for training called NEFTune that adds noise to embedding vectors, which results in better generalization (of autoregressive LMs on instruct tasks)
summarization models have very beeg generalization problem. huge
idea: let's try this technique for summarization fine-tunes and see if it helps!

tldr

dataset

lay_plos          20789
multi_news        11708
big_patent         4164
gov_report         3514
summ_screen_fd     3449
billsum            2541
lay_elife          2528
booksum            2383
cnn_dailymail      1705
stacksmol           450
qmsum               396
squality            200
xlsum_en            118
worldbank            90
narrativeqa          49
dialogsum             3
Name: subset, dtype: int64

DatasetDict({
    train: Dataset({
        features: ['text', 'summary', 'subset'],
        num_rows: 54087
    })
    validation: Dataset({
        features: ['text', 'summary', 'subset'],
        num_rows: 4262
    })
    test: Dataset({
        features: ['text', 'summary', 'subset'],
        num_rows: 4202
    })
})

Section 1

eval

train

Run Comparer