Galactica: Open-Source 120B-Param Scientific Language Model By Papers with Code, Meta AI
Papers with Code and Meta AI have published a new large language model called Galactica, this time with an edge on scientific understanding and reasoning.
Created on November 15|Last edited on November 29
Comment
This demo has been paused by the authors, but is available to researchers.
💡
A new open-source large language model has been announced by Papers with Code today called Galactica.
This language model specializes in all things math and science, being able to summarize scientific papers, generate wiki articles, answer questions, and much more. It is also fully open-source with downloadable weights for different model sizes and instructions to run it on your own hardware.
Galactica's scientific prowess
Galactica's scientific understanding comes from the work put into creating the dataset it trained on. The dataset that was used was created by tokenizing information from a variety of scientific sources into a massive dataset of tokenized information.
To ensure optimal learning of the various modalities that occurred (eg. natural language vs. math formulas vs. molecular sequences, etc.), special tokenization steps were taken, including things like identifying math operation characters or marking the start and end of different types of sequences.
Additionally, an extra special token was implemented to identify sections of step-by-step reasoning, which encourages Galactica to apply an internal working memory of sorts, which it would otherwise not be able to do.
In their testing, the researchers found their models outperformed comparable models (OPT, BLOOM, GPT-3, and others depending on the task) on many benchmarking tests, and came close to matching the top models where it's not the top performer.
For more specifics on how Galactica's tokenization approach for scientific understanding, as well as its other comparisons and benchmarks, check out the full research paper here: https://galactica.org/static/paper.pdf
Use Galactica for yourself
Galactica, created by the people at Paper with Code and Meta AI, is of course completely open-source.
The steps to use Galactica are also quite simple, not much more than a pip install and 3 lines of python code to load a model and give a prompt. You can find out how to use Galactica on the GitHub repository here: https://github.com/paperswithcode/galai
Galactica comes in 5 sizes from 250 million parameters to 120 billion parameters.
If you would rather try out Galactica without getting a python environment up and running, there's a free-to-use demo interface on Galactica's web page.
Find out more
You can read the full paper for all the details on Galactica, or head the the GitHub repository for instructions on using it for yourself.
Add a comment
Tags: ML News
Iterate on AI agents and models faster. Try Weights & Biases today.