Key Features

1. Multilingual Code Generation

   CodeGeeX excels at producing executable code in several widely-used programming languages, including Python, C++, Java, and JavaScript. Its performance remains robust across various languages.

2. Cross-Lingual Code Translation

   The model can accurately translate code snippets between different programming languages with just one click.

3. Customizable Programming Assistant

   As a VS Code extension, CodeGeeX offers features like code completion, explanations, and summarization, serving as a versatile programming aid.

4. Open-Source and Cross-Platform

   The model’s code and weights are publicly accessible, making it compatible with both Ascend and NVIDIA platforms.

HumanEval-X Benchmark

To improve the assessment of multilingual code generation, the developers introduced the HumanEval-X benchmark, featuring 820 expertly crafted coding challenges across five languages. Each challenge includes tests and solutions. CodeGeeX has demonstrated superior average performance when compared to other open-source multilingual models.

Architecture

CodeGeeX employs a transformer-based architecture, utilizing a left-to-right autoregressive decoder. It processes both code and natural language, predicting the likelihood of the next token. With 40 transformer layers, it has a hidden size of 5,120 and feed-forward layers of 20,480, resulting in a total of 13 billion parameters. The model can handle sequences up to 2,048 tokens in length.

Code Corpus

The training data for CodeGeeX includes two components. The first part comes from open-source code datasets like The Pile and CodeParrot. The Pile features code from GitHub repositories with over 100 stars, spanning 23 programming languages. Additional data is sourced from public GitHub repositories, adhering to specific quality criteria.

Training

CodeGeeX, implemented in Mindspore 1.7, was trained on 1,536 Ascend 910 AI Processors (32GB each). The training utilized mixed precision (FP16) for most layers, except for layer normalization and softmax, which used FP32. The training process lasted approximately two months, utilizing about 850 billion tokens.

Let’s Try CodeGeeX on VS Code

I initially tested CodeGeeX in VS Code due to the ease of installing the extension. Subsequently, I also tried it in the terminal.

Basic Usage

After installing the extension, follow the prompts to register and log in, or select "CodeGeeX - CodeGeeX: Login" from the right-click menu in VS Code to complete the process.

1. Code Completion/Generation:

   • Stealth Mode: CodeGeeX generates code as you pause typing.
   • Interactive Mode: Activate by pressing Ctrl+Enter to see multiple candidate suggestions.

2. Ask CodeGeeX:

   In the sidebar, you can ask questions about your code, and it will provide insights based on your selections.

The first video showcases the CodeGeeX4 model, demonstrating its capabilities and features in multilingual code generation.

Code Translation

In the CodeGeeX sidebar, you can easily translate code into a desired programming language by selecting the Translation tab and inserting the translated result directly into your editor.

Getting Started with CodeGeeX in Terminal

CodeGeeX is available for implementation in Mindspore and offers a torch-compatible version for GPU usage.

Installation Requirements:

• Python 3.7+
• CUDA 11+
• PyTorch 1.10+
• DeepSpeed 0.6+

You can install CodeGeeX through the following commands:

git clone [email protected]:THUDM/CodeGeeX.git

cd CodeGeeX

pip install -e .

Alternatively, you can use the CodeGeeX Docker image for quick setup.

Model Weights

You can apply for and download model weights via the provided link. Ensure you have sufficient disk space for the approximately 26GB download.

Inference on GPUs

To generate your first program with CodeGeeX, set the model weights path in the configuration file and write your prompt in a text file.

Quick Start for CodeGeeX2

Using transformers, you can quickly call CodeGeeX2-6B with the following code snippet:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("THUDM/codegeex2-6b", trust_remote_code=True)

model = AutoModel.from_pretrained("THUDM/codegeex2-6b", trust_remote_code=True, device='cuda')

prompt = "# language: Pythonn# write a bubble sort functionn"

inputs = tokenizer.encode(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(inputs, max_length=256, top_k=1)

response = tokenizer.decode(outputs[0])

print(response)