AlphaCode: DeepMind's Code-Competition AI Solves Problems At Human-Level Competency
DeepMind's AlphaCode AI model is being highlighted in Science Magazine this week. Its training on code-competitions led to its progress towards human-level code generation and problem solving.
Created on December 9|Last edited on December 10
Comment
8DeepMind's model for code generation, AlphaCode, first shown off in February 2022, is getting a big highlight in Science Magazine this week. AlphaCode achieves human-level problem solving skills and code writing ability as shown by performance in programming competitions.
AlphaCode's research paper is now published in Science magazine, though it's behind a pay wall. The original paper release from February is still available on ArXiv for free.
How does AlphaCode work?
The core of AlphaCode is a pretty typical encoder-decoder transformer model, starting with natural language input and ending with programming language output. This is a good choice because it follows the format for coding competitions: solving a complex word problem with code.
Training AlphaCode
AlphaCode went through two training steps: Pre-training with general code samples collected from GitHub, then fine-tuned with a custom made dataset of problem-solution pairs mimicking a code contest format.
- The GitHub code was collected from public open-source GitHub repositories and covers a wide variety of programming languages. This step teaches AlphaCode general coding principles and gets it to the point where it can write code and solve very basic problems.
- The fine-tuning dataset curated for AlphaCode, called CodeContests, consists of problems written in natural language paired with solutions written in programming languages. This format helps AlphaCode get better at solving highly complex problems and understand the competition format better. Details and downloads for the open-source CodeContests dataset can be found on GitHub.
How AlphaCode generates solutions
AlphaCode's base model by itself wouldn't be very reliable at creating good answers, so it employs a few extra steps to make sure its output is competent. It's a similar process to many other language generation models, in which the model generates many samples which are filtered down to the best output.
In AlphaCode's case, it wants to output 10 different answers to any given competition problem (following a standard submission count limit for some coding competitions). This is the basic pipeline:
- Mass Generation: First, the base model generates many raw outputs of varying (mostly bad) quality to answer the problem. These code snippets are written in C++, Python, and Java.
- Filtering: Next, the outputs are tested through a number of filters, such as checking whether they're valid code or not, or if they can pass the examples provided with a problem.
- Another model was developed (using the pre-trained model checkpoint as a starting point) which could generate new examples. It was found that whether the examples were valid or not, its use overall pushed AlphaCode towards success.
- Clustering: After filtering, AlphaCode is left with a few thousand potential solutions. Instead of just choosing 10 solutions randomly to output, the remaining solutions are clustered together by their similarity. One solution per cluster is selected for output, starting with the largest cluster. This maximizes variety, consistency, and, hopefully, success rate.
AlphaCode's human-level performance
AlphaCode was put to the test by simulating entry into 10 recent competitions hosted by Codeforces, an online code competition site.
While not a prize-winner by any means, AlphaCode was able to hit the success level of a median competitor, so ranking in the top 50% or so of competitors. AlphaCode still, however, remains at the top of code-competition AI models. AlphaCode can problem solve better than most code-writing AIs because of it's large competition-optimized training data and architecture.
Find out more
Add a comment
Tags: ML News
Iterate on AI agents and models faster. Try Weights & Biases today.