Credit: NicoElNino/Shutterstock
Meta has released a free and open large language model for generating code, dubbed Code Llama, based on its earlier released Llama 2 foundation model.
“Code Llama has the potential to be used as a productivity and educational tool to help programmers write more robust, well-documented software,” the company wrote in a blog post.
The new LLM, according to the company, was developed by further training Llama 2 on code-specific datasets (about 500 billion tokens of code and code-related data), giving it the ability to generate code and natural language about code, both from code and natural language prompts or queries.
Code Llama can also be used for code completion and debugging, Meta said, adding that it supports popular programming languages such as Python, C++, Java, PHP, Typescript, C#, and Bash among others.
Meta said it will be releasing three sizes for the new LLM with seven billion, 13 billion, and 34 billion parameters respectively.
“The 7B and 13B base and instruct models have also been trained with fill-in-the-middle (FIM) capability, allowing them to insert code into existing code, meaning they can support tasks like code completion right out of the box,” the company wrote in the blog post, adding that these three sizes address different latency requirements.
While the smaller models are faster and more suitable for low latency tasks, the 34 billion parameter model returns the best results for coding assistance, the company said, adding that the 7 billion parameter model can be served on a single GPU.
Meta is also introducing two additional variations of Code Llama — Code Llama-Python and Code-Llama Instruct.
“Code Llama – Python is a language specialised variation of Code Llama, further fine-tuned on 100B tokens of Python code,” the company wrote, adding that the specialised model was released to support the programming language’s growing popularity in the AI community.
On the other hand, Code Llama-Instruct is a fine-tuned model for natural language generation.
“The model is fed a natural language instruction input and the expected output. This makes it better at understanding what people expect out of their prompts,” the company wrote.
The generative AI-based, code-generating tool segment has seen a lot of activity in the last few months with major technology vendors such as Microsoft, Google, IBM, and AWS all releasing code-generating tools underpinned by similar foundation models.
While AWS and Google have released Amazon CodeWhisperer and Duet AI for code generation tasks, Microsoft has released its GitHub Copilot for the same. IBM, too, has released its Code Assistant for generating code from natural language prompts.
Earlier this week, IBM also previewed a specific tool in the works for refactoring COBOL code into Java for its Z mainframe systems.
