---
title: mixtral-7b-8expert
canonical_url: "https://www.modelscope.cn/models/AI-ModelScope/mixtral-7b-8expert"
md_url: "https://www.modelscope.cn/models/AI-ModelScope/mixtral-7b-8expert.md"
repository: AI-ModelScope/mixtral-7b-8expert
last_updated: 2023-12-11
license: apache-2.0
pipeline_tag: text-to-image-synthesis
tasks:
  - text-to-image-synthesis
model_type:
  - mistral
architectures:
  - MixtralForCausalLM
library_name:
  - pytorch
frameworks:
  - Pytorch
inference_backends:
  - "deploy_task text/emb"
  - "lmdeploy 0.9.1"
  - "lmdeploy_turbomind 0.9.1"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 416
stars: 4
---

# mixtral-7b-8expert

> mixtral-7b-8expert - AI-ModelScope 在 ModelScope 开源的模型。This is a preliminary HuggingFace implementation of the newly released MoE model by MistralAi. Make sure to load with trustremotecode=True.

AI-ModelScope/mixtral-7b-8expert 是 ModelScope 魔搭社区上的text-to-image-synthesis模型，采用 apache-2.0 许可，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。

- **Repository**: AI-ModelScope/mixtral-7b-8expert
- **License**: apache-2.0
- **Tasks**: text-to-image-synthesis
- **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2
- **Downloads**: 416
- **Stars**: 4
- **Last updated**: 2023-12-11

Source: https://www.modelscope.cn/models/AI-ModelScope/mixtral-7b-8expert

---

# Mixtral 7b 8 Expert

![image/png](https://cdn-uploads.huggingface.co/production/uploads/62e3b6ab0c2a907c388e4965/6m3e2d2BNXDjy6_qHd2LT.png)

This is a preliminary HuggingFace implementation of the newly released MoE model by MistralAi. Make sure to  load with `trust_remote_code=True`.

Thanks to @dzhulgakov for his early implementation (https://github.com/dzhulgakov/llama-mistral) that helped me find a working setup.

Also many thanks to our friends at [LAION](https://laion.ai) and [HessianAI](https://hessian.ai/) for the compute used for these projects!

Benchmark scores:
```
hella swag: 0.8661
winogrande: 0.824
truthfulqa_mc2: 0.4855
arc_challenge:  0.6638
gsm8k: 0.5709
MMLU: 0.7173
```

# Basic Inference setup

```python
# transformers>=4.36 (build from source)
import torch
from modelscope import AutoModelForCausalLM, AutoTokenizer, snapshot_download

model = AutoModelForCausalLM.from_pretrained('AI-ModelScope/mixtral-7b-8expert', low_cpu_mem_usage=True, 
                                             device_map="auto", trust_remote_code=True)
tok = AutoTokenizer.from_pretrained('AI-ModelScope/mixtral-7b-8expert')
x = tok.encode("The mistral wind in is a phenomenon ", return_tensors="pt").cuda()
x = model.generate(x, max_new_tokens=128).cpu()
print(tok.batch_decode(x))
```

# Conversion

Use `convert_mistral_moe_weights_to_hf.py --input_dir ./input_dir --model_size 7B --output_dir ./output` to convert the original consolidated weights to this HF setup.

Come chat about this in our [Disco(rd)](https://discord.gg/S8W8B5nz3v)! :)