Stable Code 3B: Stability's New Compact Code Completion LLM
Known for their image generation models, Stability looks to enter the market with a new compact LLM Coding Model
Created on January 17|Last edited on January 17
Comment
Stable Code 3B, a new Large Language Model (LLM) specializing in code completion, has been introduced as a significant upgrade over its predecessor, Stable Code Alpha 3B. This model stands out for its compact size, high efficiency, and broad programming language support, making it a noteworthy tool for developers.
Model Overview
Stable Code 3B, built upon the Stable LM 3B foundational model, which was trained on 4 trillion tokens of natural language data, distinguishes itself by its size and capability. Despite being 60% smaller than CodeLLaMA 7b, a well-known model in the coding community, Stable Code 3B delivers comparable performance. A key feature of this model is its ability to operate on standard laptops, including models without a dedicated GPU, like the MacBook Air, offering offline functionality.
Technical Enhancements
One of the model's highlights is its capacity for 'Fill in the Middle' (FIM) capabilities and an expanded context size. Stable Code is initially trained on sequences up to 16,384 tokens, but can extend its context length up to 100,000 tokens. This flexibility is made possible by the implementation of Rotary Embeddings, a technique also used in CodeLlama.
Language Proficiency and Performance
Stable Code 3B supports 18 programming languages, chosen based on popularity in the 2023 StackOverflow Developer Survey. It shows remarkable performance on the MultiPL-E metrics, particularly notable in models of its size.
Training Process
The model's development involved a multi-stage training process, starting with a pre-trained natural language model, followed by unsupervised fine-tuning on various coding datasets. These include CommitPack, GitHub Issues, StarCoder, and math-related datasets. The model also incorporates Flash Attention 2, enhancing its efficiency.
Commercial Applications
Stable Code 3B is part of the new Stability AI Membership, featuring in their commercial Core Model offerings alongside SDXL Turbo and Stable Video Diffusion. Interested users can explore these offerings on the Stability AI Membership page.
Engaging with the Community
For those interested in delving deeper into Stable Code 3B, a full technical report with detailed insights and model evaluations is planned for release. The company encourages transparency and community involvement, inviting users to stay informed via their newsletter and to engage through various social media platforms like Twitter, Instagram, LinkedIn, and their Discord community.
This release marks a significant step in the evolution of coding tools, offering an accessible yet powerful solution for a wide range of programming needs.
Add a comment
Tags: ML News
Iterate on AI agents and models faster. Try Weights & Biases today.