Surveying Survey Papers on the LLM Era

Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond



Practical Guide for Models

decoder-only models have been growing tremendously in popularity with encoder-only models fading away
OpenAI is dominant in the LLM industry
Meta is consistently open-sourcing their LLMs and LLM work
the introduction of GPT-3 led to more closed-source models, potentially for commercial use
encoder-decoder models may be promising



BERT-style: Masked Language Modeling (MLM) where an LM is trained to fill-in-the-blank of a sentence (certain words are masked in a sentence)
GPT-style: Autoregressive LMs boomed with GPT-3

Practical Guide for Data

LLMs > fine-tuned LMs (the authors consider models <20B parameters to be not LLMs) on out-of-distribution data
LLMs > fine-tuned LMs if limited annotated data; if not limited annotated data, then either can work depending on the objective and factors at hand
advisable to pick models pretrained on similar data for downstream tasks (transfer learning-esque)

no annotated data: using an LLM zero-shot is most suitable
few annotated data: using LLM with in-context learning; though techniques exist where LMs (not LLMs) can be finetuned on minimal data
abundant annotated data: finetuned LMs are viable but, of course, LLMs are also just as viable here (though there are pros and cons for each that aren't about data availability)



Practical Guide for NLP Tasks

sentiment analysis: finetuned LMs = LLMs on IMDB and SST
toxicity detection: finetuned LMs > LLMs on CivilComments
(NLI) RTE and SNLI: finetuned LMs > LLMs, LLMs = finetuned LMs on CB
QA: finetuned LMs > LLMs on SQuADv2 and QuAC
NER: finetuned LMs >> LLMs CoNLL03

converting input texts to new symbol sequences (task-specific generation like summarizing a paragraph; the model only generates summaries)
open-ended generation

LLMs are competent in machine translation (MT) and their performance can be further improved by including more multilingual data in their pretraining
rich resource MT scenarios, the finetuned LMs slightly beat LLMs
low resource MT scenarios, finetuned LMs significantly beat LLMs

LLMs are strong in knowledge-intensive tasks
they falter when the knowledge doesn't align with their learned knowledge or when contextual knowledge is needed

Closed-book QA: NaturalQuestions, WebQuestions, and TriviaQA where LLMs beat finetuned LMs
however, in cases where the dataset works against the knowledge the model knows (or the LLM simply hasn't learned that knowledge yet) the model falls behind finetuned LMs
an alternative is retrieval augmentation which is essentially a "memory bank" database for the model to retrieve information from

with exponential scaling, models grow more adept in commonsense and arithmetic reasoning
emergent abilities
in some cases, scaling does not improve the model's performance

Finetuned models are relevant in tasks different from the pretraining of LLMs
LLMs can be used as quality evaluation in NLP tasks

LLMs >> finetuned LMs in real-world but evaluating LLMs is still an open problem

if cost and strict latency are issues, finetuned models with PEFT (parameter-efficient finetuning) are better
LLMs still demonstrate a bit of shortcutting issues when it comes to task-specific datasets
safety concerns in LLMs are a top priority as their output is open-ended

GPT-1 sits at 117M parameters, GPT-2 at 1.5B, and GPT-3 at 175B parameters
training 11B T5 model costed $1.3M for a single run
single training run for GPT-3 175B costed about $4.6M
PaLM consumed 3.4 GWh in about 2 months
OpenAI partnered with Microsoft and Azure, leveraging their immense supercomputer of 285k CPU cores and 10k high-end GPUs
LLMs are too large to be on a single computer so they are provided as services via APIs
PEFT can be used to train a model for a substantially smaller cost; common methods include Low Rank Adaptation (LoRA), Prefix Tuning, and P-Tuning
accuracy and robustness are correlated in LLMs but finetuning an LLM on task-specific data may lead to overparameterization and overfitting
LLMs perpetuate societal biases
LLMs are vulnerable to label bias and making shortcuts even in zero-shot scenarios
LLMs can advertently or inadvertently generate harmful content and they can hallucinate, generate false information

evaluating models on both benchmarks and real-world datasets
model and safety alignment problem with LLMs
predicting performance scaling

A Bibliometric Review of Large Language Models Research from 2017 to 2023





encoding titles and abstracts as embedding vectors using SBERT
reducing the dimensionality with UMAP
clustering with k-Means
Bag of Words (BoW) for counting frequency
c-TF-IDF (class-based TF-IDF) to extract difference of topical keywords

Algorithm and NLP tasks
Medical and Engineering Applications
Social and Humanitarian Applications
Critical Studies
Infrastructures

Publication frequency exploded since 2018 and especially so with ChatGPT. Most publications are for algorithms and NLP tasks.

There are no obviously distinct clusters. Infrastructure focuses on distributed computing and hardware and accelerators.

(a) show general NLP concepts and tasks. (b) seems to focus on pretraining and NLP techniques. (c) focuses on articles like fake news and hate speech.

Most papers published in North America, Asia-Pacific, and Europe. The authors found that USA and China are at the forefront of algorithms and NLP tasks research.

Universities play a large role in LLM research but large tech companies have grown to become significant contributors in their own right.

Conclusion

References

Also check out this blog post where I cover another survey paper!