---
title: TurboSparse-Mixtral-Instruct
canonical_url: "https://www.modelscope.cn/models/PowerInfer/TurboSparse-Mixtral-Instruct"
md_url: "https://www.modelscope.cn/models/PowerInfer/TurboSparse-Mixtral-Instruct.md"
repository: PowerInfer/TurboSparse-Mixtral-Instruct
last_updated: 2024-12-31
license: apache-2.0
pipeline_tag: feature-extraction
tasks:
  - feature-extraction
model_type:
  - turbosparsemixtral
architectures:
  - TurboSparseMixtralForCausalLM
parameters: 51.8B
tensor_type:
  - BF16
library_name:
  - safetensors
  - pytorch
frameworks:
  - pytorch
language:
  - en
downloads: 5874
stars: 0
---

# TurboSparse-Mixtral-Instruct

> TurboSparse-Mixtral-Instruct - PowerInfer 在 ModelScope 开源的模型。Model Card for TurboSparse-Mixtral The TurboSparse-Mixtral Large Language Model (LLM) is an sparsified version of the Mixtral.

PowerInfer/TurboSparse-Mixtral-Instruct 是 ModelScope 魔搭社区上的 51.8B 参数feature-extraction模型，采用 apache-2.0 许可。

- **Repository**: PowerInfer/TurboSparse-Mixtral-Instruct
- **License**: apache-2.0
- **Tasks**: feature-extraction
- **Parameters**: 51.8B
- **Downloads**: 5874
- **Stars**: 0
- **Last updated**: 2024-12-31

Source: https://www.modelscope.cn/models/PowerInfer/TurboSparse-Mixtral-Instruct

---

# Model Card for TurboSparse-Mixtral
The [TurboSparse-Mixtral](https://arxiv.org/abs/2406.05955) Large Language Model (LLM) is an sparsified version of the Mixtral.

<img src="takeaway.png" alt="avatar" width="300" height="200"/>

The average performance is evaluated using benchmarks from the OpenLLM Leaderboard.

## Inference

Our code for accelerating TurboSparse-Mixtral is currently being refined. Stay tuned! Now you can run this model like dense model.

## Chat-Template

During sparsification, we also utilize some SFT datasets.
We take ChatML as our chat template:
```
<|im_start|>user\n{{content}}<|im_end|>\n<|im_start|>assistant\n
```

## Allow Finetuning

As we merged the predictors for FFN neurons in models, you can finetune TurboSparse-Mixtral with any framework and algorithm.

## Limitations
* TurboSparse, having just undergone training with 150B tokens, may still exhibit performance gaps in certain tasks.
* The TurboSparse model has only been trained on English-language datasets, hence its capabilities in other languages are still lacking.
* The model may produce unexpected outputs due to its small size, limited training tokens and probabilistic generation paradigm.

## License

The model is licensed under Apache-2.0, while model weights are fully open for academic research and also allow **free** commercial usage.