---
title: CodeFuse-13B
canonical_url: "https://www.modelscope.cn/models/codefuse-ai/CodeFuse-13B"
md_url: "https://www.modelscope.cn/models/codefuse-ai/CodeFuse-13B.md"
repository: codefuse-ai/CodeFuse-13B
last_updated: 2023-10-25
license: other
model_type:
  - gpt_neox
architectures:
  - GPTNeoXForCausalLM
library_name:
  - pytorch
frameworks:
  - pytorch
inference_backends:
  - "deploy_task text/emb"
  - "vllm 0.9.2"
downloads: 125811
stars: 38
---

# CodeFuse-13B

> CodeFuse-13B - codefuse-ai 在 ModelScope 开源的模型。CodeFuse-13B是基于GPT-NeoX框架训练的13B参数代码生成模型，能够处理4096个字符的代码序列。该模型在1000B Token的代码、中文、英文数据数据集上进行预训练，覆盖超过40种编程语言。为了进一步提升生成代码的效果和质量，该模型还在CodeFuse-Evol-instruction-66k数据集上进行了微调，使得该模型能够生成更加准确、高效、符合要求的代码。在HumanEval评测集上Pass@1达…

codefuse-ai/CodeFuse-13B 是 ModelScope 魔搭社区上的机器学习模型，采用 other 许可，可用 deploy_task text/emb、vllm 0.9.2 部署。

- **Repository**: codefuse-ai/CodeFuse-13B
- **License**: other
- **Inference backends**: deploy_task text/emb, vllm 0.9.2
- **Downloads**: 125811
- **Stars**: 38
- **Last updated**: 2023-10-25

Source: https://www.modelscope.cn/models/codefuse-ai/CodeFuse-13B

---

# Model Card for CodeFuse-13B

<p align="center">
    <img src="https://modelscope.cn/api/v1/models/codefuse-ai/CodeFuse-QWen-14B/repo?Revision=master&FilePath=LOGO.jpg&View=true" width="800"/>
<p>


[[中文]](#chinese)    [[English]](#english)

<a id="english"></a>


## Model Description
CodeFuse-13B is a 13 billion parameter code generation model trained on the GPT-NeoX framework, capable of handling code sequences of up to 4096 characters. This model was pretrained on a dataset consisting of 1000B token code, Chinese, and English data, covering over 40 programming languages. To further enhance the effectiveness and quality of the generated code, the model was fine-tuned on the CodeFuse-Evol-instruction-66k dataset, enabling it to produce more accurate, efficient, and compliant code. Pass@1 achieved 37.1% on the HumanEval evaluation set(BeamSearch strategy, BeamSize=3).


## Code Community

**Homepage**: 🏡 https://github.com/codefuse-ai (**Please give us your support with a Star🌟 + Fork🚀 + Watch👀**)

+ If you wish to fine-tune the model yourself, you can visit ✨[MFTCoder](https://github.com/codefuse-ai/MFTCoder)✨✨

+ If you wish to deploy the model yourself, you can visit ✨[FasterTransformer4CodeFuse](https://github.com/codefuse-ai/FasterTransformer4CodeFuse)✨✨

+ If you wish to see a demo of the model, you can visit ✨[CodeFuse Demo](https://github.com/codefuse-ai/codefuse)✨✨


## Requirements
* Python 3.8 or above.
* PyTorch 1.12 or above, with a recommendation for 2.0 or above.
* Transformers 4.24.0 or above.
* It is advisable to use CUDA 11.4 or above (for GPU users and flash-attention users, this option should be considered).


## Quickstart

```python
import torch
from modelscope import AutoModelForCausalLM, AutoTokenizer, snapshot_download

model_dir = snapshot_download('codefuse-ai/CodeFuse-13B', revision='v1.0.0')
tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(model_dir, device_map="auto", torch_dtype=torch.float16).eval()

input_ids = tokenizer.encode("# language: Python\ndef quick_sort(array):\n", return_tensors="pt").to("cuda")
output_ids = model.generate(input_ids, max_new_tokens=200)

print(tokenizer.decode(output_ids[0]))
```

## MD5
We notice that the file may be corrupted during transfer process. Please check MD5 value before use.

| Model File                       | MD5 Value                        |
|:---------------------------------|:--------------------------------:|
| pytorch_model-00001-of-00006.bin | b79e4ccc93c40fa6113aaf6a434473d5 |
| pytorch_model-00002-of-00006.bin | 5a82f19e3f62c693e41fe627084c722b |
| pytorch_model-00003-of-00006.bin | d4b53c391a353d0fc0a1be1c913d5f04 |
| pytorch_model-00004-of-00006.bin | f9e3dcdea13ff02f4e3aad4f9db7a33f |
| pytorch_model-00005-of-00006.bin | 698a8f2f05723a572193733bce12eb93 |
| pytorch_model-00006-of-00006.bin | 312439d0b810f1bb81034fe094ff84c7 |

<a id="chinese"></a>

## 简介
CodeFuse-13B是基于GPT-NeoX框架训练的13B参数代码生成模型，能够处理4096个字符的代码序列。该模型在1000B Token的代码、中文、英文数据数据集上进行预训练，覆盖超过40种编程语言。为了进一步提升生成代码的效果和质量，该模型还在CodeFuse-Evol-instruction-66k数据集上进行了微调，使得该模型能够生成更加准确、高效、符合要求的代码。在HumanEval评测集上Pass@1达到37.1%(采用BeamSearch解码，其中BeamSize=3)。


## 代码社区
**大本营**： 🏡 https://github.com/codefuse-ai （**请支持我们的项目 Star🌟 + Fork🚀 + Watch👀**）

+ 如果您想自己微调该模型，可以访问 ✨[MFTCoder](https://github.com/codefuse-ai/MFTCoder)✨✨

+ 如果您想自己部署该模型，可以访问 ✨[FasterTransformer4CodeFuse](https://github.com/codefuse-ai/FasterTransformer4CodeFuse)✨✨

+ 如果您想观看该模型示例，可以访问 ✨[CodeFuse Demo](https://github.com/codefuse-ai/codefuse)✨✨



## 要求
* python 3.8及以上版本
* pytorch 1.12及以上版本，推荐2.0及以上版本
* transformers 4.24.0及以上版本
* 建议使用CUDA 11.4及以上（GPU用户、flash-attention用户等需考虑此选


## 快速使用

```python
import torch
from modelscope import AutoModelForCausalLM, AutoTokenizer, snapshot_download

model_dir = snapshot_download('codefuse-ai/CodeFuse-13B', revision='v1.0.0')
tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModelForCausalLM.from_pretrained(model_dir, device_map="auto", torch_dtype=torch.float16).eval()

input_ids = tokenizer.encode("# language: Python\ndef quick_sort(array):\n", return_tensors="pt").to("cuda")
output_ids = model.generate(input_ids, max_new_tokens=200)

print(tokenizer.decode(output_ids[0]))
```

## MD5
我们发现模型文件可能会在传输过程中损坏，使用前请检查文件MD5值。

| 模型文件                           | MD5值                            |
|:---------------------------------|:--------------------------------:|
| pytorch_model-00001-of-00006.bin | b79e4ccc93c40fa6113aaf6a434473d5 |
| pytorch_model-00002-of-00006.bin | 5a82f19e3f62c693e41fe627084c722b |
| pytorch_model-00003-of-00006.bin | d4b53c391a353d0fc0a1be1c913d5f04 |
| pytorch_model-00004-of-00006.bin | f9e3dcdea13ff02f4e3aad4f9db7a33f |
| pytorch_model-00005-of-00006.bin | 698a8f2f05723a572193733bce12eb93 |
| pytorch_model-00006-of-00006.bin | 312439d0b810f1bb81034fe094ff84c7 |