Meta releases Code Llama, a new LLM geared for programming

True to the rumors and advance reports, Meta Platforms, the company formerly known as Facebook, today unveiled Code Llama, its new generative AI large language model (LLM) designed specifically for programming — and like the more general-purpose LLaMA 2, it’s open source and licensed for commercial use.

Code Llama is “designed to support software engineers in all sectors — including research, industry, open source projects, NGOs, and businesses,” Meta says in its blog post announcing the models.

The tool immediately becomes a major rival to OpenAI’s Codex (powered by a modified GPT-3), the Codex-powered Github Copilot from Microsoft, and other coding-specific LLM assistants such as Stack Overflow’s OverflowAI.

In its blog post, Meta explains that Code LlaMA is a “code-specialized” version of LLaMA 2 that can generate code, complete code, create developer notes and documentation, be used for debugging, and more. It supports Python, C++, Java, PHP, Typescript (Javascript), C# and Bash. You can read the full research paper from Meta about its performance here, which describes Code LlaMA as a “family” of LLMs for code.

Building on that analogy, the family includes three main members: a 7-billion, a 13-billion and a 34-billion parameter model, each trained on 500 billion tokens. The smaller models are designed to run on fewer GPUs (the 7-billion model can run on a single one), a beneficial attribute given the rumored scarcity in this critical piece of hardware at the moment, and Meta says both are faster than its 34-billion big model.

All models support up to 100,000 tokens for their prompts. This means “users can provide the model with more context from their codebase to make the generations more relevant,” according to Meta.

The LLaMA extended family also includes two fine-tuned models, one for Python and one for Instruct, the latter of which “has [been] fine-tuned to generate helpful and safe answers in natural language,” and therefore, Meta says, should be used when generating new code from natural language prompts. That is, it returns safer, more expected and perhaps less creative responses.

You can download Code LlaMA directly from Meta here and find the source code on Github here.