---
title: DeepSeek-Coder-V2-Lite-Instruct-GGUF
canonical_url: "https://www.modelscope.cn/models/second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF"
md_url: "https://www.modelscope.cn/models/second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF.md"
repository: second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF
last_updated: 2024-10-28
license: other
pipeline_tag: text-generation
tasks:
- text-generation
model_type:
- deepseek
architectures:
- DeepseekV2ForCausalLM
base_model:
- deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct
base_model_relation: quantized
library_name:
- gguf
- pytorch
frameworks:
- pytorch
inference_backends:
- "deploy_task text/emb"
- "lmdeploy 0.9.1"
- "lmdeploy_turbomind 0.9.1"
- "sglang 0.5.2"
- "vllm 0.9.2"
downloads: 2671
stars: 0
tags:
- gguf
---
# DeepSeek-Coder-V2-Lite-Instruct-GGUF
> DeepSeek-Coder-V2-Lite-Instruct-GGUF - second-state 在 ModelScope 开源的模型。DeepSeek-Coder-V2-Lite-Instruct-GGUF
second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF 是 ModelScope 魔搭社区上的text-generation模型，采用 other 许可，基于 deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct 构建，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。
- **Repository**: second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF
- **License**: other
- **Tasks**: text-generation
- **Base model**: deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct
- **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2
- **Tags**: gguf
- **Downloads**: 2671
- **Stars**: 0
- **Last updated**: 2024-10-28
Source: https://www.modelscope.cn/models/second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF
---
# DeepSeek-Coder-V2-Lite-Instruct-GGUF
## Original Model
[deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct](https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct)
## Run with LlamaEdge
- LlamaEdge version: [v0.12.3](https://github.com/LlamaEdge/LlamaEdge/releases/tag/0.12.3) and above
- Prompt template
- Prompt type: `deepseek-chat-2`
- Prompt string
```text
<|begin_of_sentence|>{system_message}
User: {user_message_1}
Assistant: {assistant_message_1}<|end_of_sentence|>User: {user_message_2}
Assistant:
```
- Context size: `128000`
- Run as LlamaEdge service
```bash
wasmedge --dir .:. \
--nn-preload default:GGML:AUTO:DeepSeek-Coder-V2-Lite-Instruct-Q5_K_M.gguf \
llama-api-server.wasm \
--prompt-template deepseek-chat-2 \
--ctx-size 128000 \
--model-name DeepSeek-Coder-V2-Lite-Instruct
```
- Run as LlamaEdge command app
```bash
wasmedge --dir .:. \
--nn-preload default:GGML:AUTO:DeepSeek-Coder-V2-Lite-Instruct-Q5_K_M.gguf \
llama-chat.wasm \
--prompt-template deepseek-chat-2 \
--ctx-size 128000
```
## Quantized GGUF Models
| Name | Quant method | Bits | Size | Use case |
| ---- | ---- | ---- | ---- | ----- |
| [DeepSeek-Coder-V2-Lite-Instruct-Q2_K.gguf](https://huggingface.co/second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF/blob/main/DeepSeek-Coder-V2-Lite-Instruct-Q2_K.gguf) | Q2_K | 2 | 6.43 GB| smallest, significant quality loss - not recommended for most purposes |
| [DeepSeek-Coder-V2-Lite-Instruct-Q3_K_L.gguf](https://huggingface.co/second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF/blob/main/DeepSeek-Coder-V2-Lite-Instruct-Q3_K_L.gguf) | Q3_K_L | 3 | 8.46 GB| small, substantial quality loss |
| [DeepSeek-Coder-V2-Lite-Instruct-Q3_K_M.gguf](https://huggingface.co/second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF/blob/main/DeepSeek-Coder-V2-Lite-Instruct-Q3_K_M.gguf) | Q3_K_M | 3 | 8.13 GB| very small, high quality loss |
| [DeepSeek-Coder-V2-Lite-Instruct-Q3_K_S.gguf](https://huggingface.co/second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF/blob/main/DeepSeek-Coder-V2-Lite-Instruct-Q3_K_S.gguf) | Q3_K_S | 3 | 7.49 GB| very small, high quality loss |
| [DeepSeek-Coder-V2-Lite-Instruct-Q4_0.gguf](https://huggingface.co/second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF/blob/main/DeepSeek-Coder-V2-Lite-Instruct-Q4_0.gguf) | Q4_0 | 4 | 8.91 GB| legacy; small, very high quality loss - prefer using Q3_K_M |
| [DeepSeek-Coder-V2-Lite-Instruct-Q4_K_M.gguf](https://huggingface.co/second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF/blob/main/DeepSeek-Coder-V2-Lite-Instruct-Q4_K_M.gguf) | Q4_K_M | 4 | 10.4 GB| medium, balanced quality - recommended |
| [DeepSeek-Coder-V2-Lite-Instruct-Q4_K_S.gguf](https://huggingface.co/second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF/blob/main/DeepSeek-Coder-V2-Lite-Instruct-Q4_K_S.gguf) | Q4_K_S | 4 | 9.53 GB| small, greater quality loss |
| [DeepSeek-Coder-V2-Lite-Instruct-Q5_0.gguf](https://huggingface.co/second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF/blob/main/DeepSeek-Coder-V2-Lite-Instruct-Q5_0.gguf) | Q5_0 | 5 | 10.8 GB| legacy; medium, balanced quality - prefer using Q4_K_M |
| [DeepSeek-Coder-V2-Lite-Instruct-Q5_K_M.gguf](https://huggingface.co/second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF/blob/main/DeepSeek-Coder-V2-Lite-Instruct-Q5_K_M.gguf) | Q5_K_M | 5 | 11.9 GB| large, very low quality loss - recommended |
| [DeepSeek-Coder-V2-Lite-Instruct-Q5_K_S.gguf](https://huggingface.co/second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF/blob/main/DeepSeek-Coder-V2-Lite-Instruct-Q5_K_S.gguf) | Q5_K_S | 5 | 11.1 GB| large, low quality loss - recommended |
| [DeepSeek-Coder-V2-Lite-Instruct-Q6_K.gguf](https://huggingface.co/second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF/blob/main/DeepSeek-Coder-V2-Lite-Instruct-Q6_K.gguf) | Q6_K | 6 | 14.1 GB| very large, extremely low quality loss |
| [DeepSeek-Coder-V2-Lite-Instruct-Q8_0.gguf](https://huggingface.co/second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF/blob/main/DeepSeek-Coder-V2-Lite-Instruct-Q8_0.gguf) | Q8_0 | 8 | 16.7 GB| very large, extremely low quality loss - not recommended |
| [DeepSeek-Coder-V2-Lite-Instruct-f16.gguf](https://huggingface.co/second-state/DeepSeek-Coder-V2-Lite-Instruct-GGUF/blob/main/DeepSeek-Coder-V2-Lite-Instruct-f16.gguf) | f16 | 16 | 31.4 GB| |
*Quatized with llama.cpp b3212*
