---
title: WizardLM-2-8x22B-GGUF
canonical_url: "https://www.modelscope.cn/models/AI-ModelScope/WizardLM-2-8x22B-GGUF"
md_url: "https://www.modelscope.cn/models/AI-ModelScope/WizardLM-2-8x22B-GGUF.md"
repository: AI-ModelScope/WizardLM-2-8x22B-GGUF
last_updated: 2024-04-18
pipeline_tag: text-generation
tasks:
  - text-generation
library_name:
  - gguf
  - pytorch
frameworks:
  - pytorch
downloads: 3511
stars: 1
tags:
  - quantized
  - 2-bit
  - 3-bit
  - 4-bit
  - 5-bit
  - 6-bit
  - 8-bit
  - GGUF
  - transformers
  - safetensors
  - mistral
  - text-generation
  - "arxiv:2304.12244"
  - "arxiv:2306.08568"
  - "arxiv:2308.09583"
  - "license:apache-2.0"
  - autotrain_compatible
  - endpoints_compatible
  - text-generation-inference
  - "region:us"
  - gguf
---

# WizardLM-2-8x22B-GGUF

> WizardLM-2-8x22B-GGUF - AI-ModelScope 在 ModelScope 开源的模型。MaziyarPanahi/WizardLM-2-8x22B-GGUF Model creator: microsoft Original model: microsoft/WizardLM-2-8x22B

- **Repository**: AI-ModelScope/WizardLM-2-8x22B-GGUF
- **Tasks**: text-generation
- **Tags**: quantized, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, 8-bit, GGUF, transformers, safetensors, mistral, text-generation, arxiv:2304.12244, arxiv:2306.08568, arxiv:2308.09583, license:apache-2.0, autotrain_compatible, endpoints_compatible, text-generation-inference, region:us, gguf
- **Downloads**: 3511
- **Stars**: 1
- **Last updated**: 2024-04-18

Source: https://www.modelscope.cn/models/AI-ModelScope/WizardLM-2-8x22B-GGUF

---

# [MaziyarPanahi/WizardLM-2-8x22B-GGUF](https://huggingface.co/MaziyarPanahi/WizardLM-2-8x22B-GGUF)
- Model creator: [microsoft](https://huggingface.co/microsoft)
- Original model: [microsoft/WizardLM-2-8x22B](https://huggingface.co/microsoft/WizardLM-2-8x22B)

## Description
[MaziyarPanahi/WizardLM-2-8x22B-GGUF](https://huggingface.co/MaziyarPanahi/WizardLM-2-8x22B-GGUF) contains GGUF format model files for [microsoft/WizardLM-2-8x22B](https://huggingface.co/microsoft/WizardLM-2-8x22B).

## How to download
You can download only the quants you need instead of cloning the entire repository as follows:


```
huggingface-cli download MaziyarPanahi/WizardLM-2-8x22B-GGUF --local-dir . --include '*Q2_K*gguf'
```

On Windows:

```sh
huggingface-cli download MaziyarPanahi/WizardLM-2-8x22B-GGUF --local-dir . --include *Q4_K_S*gguf
```

## Load sharded model

`llama_load_model_from_file` will detect the number of files and will load additional tensors from the rest of files.

```sh
llama.cpp/main -m WizardLM-2-8x22B.Q2_K-00001-of-00005.gguf -p "Building a website can be done in 10 simple steps:\nStep 1:" -n 1024 -e
```

## Prompt template

```
{system_prompt}
USER: {prompt}
ASSISTANT: </s>
```

or

```
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, 
detailed, and polite answers to the user's questions. USER: Hi ASSISTANT: Hello.</s>
USER: {prompt} ASSISTANT: </s>......
```