This week in AI: Meta LLaMA 2, Meta-Transformer, StabilityAI FreeWilly
A round-up of some of the biggest and most significant changes in the world of artificial intelligence in recent weeks — from LLaMA2 to Meta-Tranformer, and beyond.
Created on July 25|Last edited on July 25
Comment
Meta LLaMA 2
Along with version 2 of their LLaMA models, Meta is continuing their partnership with Microsoft, placing LLaMA 2 in Azure's AI Model Catalog. In light of their open approach, Meta's LLaMA 2 is free for research and commercial use.
What makes these models different from the first generation of models? According to their paper, LLaMA 2 models witnessed "more robust data cleaning, updated our data mixes, trained on 40% more total tokens, doubled the context length, and used grouped-query attention (GQA) to improve inference scalability". This newer generation of LLaMA models are tailored towards conversational use cases.
Meta-Transformer
Meta-Transformer, not from Meta, is a unified, multi-modal Transformer architecture!
It's safe to say this transformer is really multi-modal, not just text and images. Their website has a great video, below, walking through their paper's method.
The overall architecture of their Meta-Transformer consists of a data-to-sequence tokenizer layer which, itself, consists of multiple modality-specific tokenizers. The tokenized input enters a shared token space which can all be fed into the unified model. The output of this unified model is fed into task-specific models.
They benchmarked their model across dozens of benchmarks and other models!
StabilityAI FreeWilly
StabilityAI recently released 2 new LLaMA variants: FreeWilly and FreeWilly2. Both of these models were trained on synthetically generated datasets, borrowing from the findings in "Orca: Progressive Learning from Complex Explanation Traces of GPT-4". They train on only 600k data points consisting of a mix of simple and complex examples, roughly 10% of the original dataset in the Orca paper while achieving exceptional performance. The FreeWilly family of LLMs is available on HuggingFace with model cards and inference APIs to go along with them.
References
Add a comment
Tags: ML News
Iterate on AI agents and models faster. Try Weights & Biases today.