Google AI Introduces PaLM: The Pathways Language Model
On April 4, 2022 Google AI announced the Pathways Language Model (PaLM) in their quest to create, "a single model that could generalize across domains and tasks while being highly efficient."
Created on April 5|Last edited on February 12
Comment
Google AI has just announced the development of the Pathways system. This 540-billion parameter model achieves state-of-the-art performance across most of the hundreds of language understanding tasks it was evaluated on.
As the scale of the model increases, the performance improves across tasks while also unlocking new capabilities.
What Is PaLM?
As they describe it, "... a 540-billion parameter, densely activated, Transformer language model ... trained on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods."
PaLM demostrates the largest TPU-based training to date, scaling on the Pathways system to 6144 chips. A significant increase in scale vs other Large language Models.
Why Do We Care?
PaLM produced, "... breakthrough capabilities on numerous very difficult tasks" including:
- Language Understanding and Generation where it out-performed previous large models on 28 of 29 tasks.
- Tested on Beyond the Imitation Game Benchmark (BIG-bench), where it showed solid performance in natural language understanding and generation capabilities.
Examples that showcase PaLM 540B 1-shot performance on BIG-bench tasks: labeling cause and effect, conceptual understanding, guessing movies from emoji, and finding synonyms and counterfactuals.
- Reasoning tasks, where is outperformed previous models by impressive margins.
PaLM explains an original joke with two-shot prompts.
- Code Generation including writing code given a natural language desciption.
Examples of a fine-tuned PaLM 540B model on text-to-code tasks, such as GSM8K-Python and HumanEval, and code-to-code tasks, such as Transcoder.
Read It For Yourself
Add a comment
Tags: ML News
Iterate on AI agents and models faster. Try Weights & Biases today.