---
title: DeepSeek-R1-Distill-Qwen-32B-MLX
canonical_url: "https://www.modelscope.cn/models/mlx-community/DeepSeek-R1-Distill-Qwen-32B-MLX"
md_url: "https://www.modelscope.cn/models/mlx-community/DeepSeek-R1-Distill-Qwen-32B-MLX.md"
repository: mlx-community/DeepSeek-R1-Distill-Qwen-32B-MLX
last_updated: 2025-03-11
pipeline_tag: text-generation
tasks:
  - text-generation
base_model:
  - deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
base_model_relation: finetune
parameters: 114.3B
tensor_type:
  - U32
  - F16
  - BF16
library_name:
  - mlx
  - safetensors
  - pytorch
frameworks:
  - pytorch
language:
  - en
downloads: 599
stars: 1
tags:
  - chat
  - mlx
  - conversations
---

# DeepSeek-R1-Distill-Qwen-32B-MLX

> DeepSeek-R1-Distill-Qwen-32B-MLX - mlx-community 在 ModelScope 开源的模型。mlx-community/DeepSeek-R1-Distill-Qwen-32B

mlx-community/DeepSeek-R1-Distill-Qwen-32B-MLX 是 ModelScope 魔搭社区上的 114.3B 参数text-generation模型，基于 deepseek-ai/DeepSeek-R1-Distill-Qwen-32B 构建。

- **Repository**: mlx-community/DeepSeek-R1-Distill-Qwen-32B-MLX
- **Tasks**: text-generation
- **Parameters**: 114.3B
- **Base model**: deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
- **Tags**: chat, mlx, conversations
- **Downloads**: 599
- **Stars**: 1
- **Last updated**: 2025-03-11

Source: https://www.modelscope.cn/models/mlx-community/DeepSeek-R1-Distill-Qwen-32B-MLX

---

# mlx-community/DeepSeek-R1-Distill-Qwen-32B

This Model [mlx-community/DeepSeek-R1-Distill-Qwen-32B](https://huggingface.co/mlx-community/DeepSeek-R1-Distill-Qwen-32B) contains multiple quantized variants of the base model [deepseek-ai/DeepSeek-R1-Distill-Qwen-32B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B). The model was converted to MLX format using mlx-lm version 0.21.5.

The conversion process applied different quantization strategies to produce variants that offer trade-offs between memory footprint, inference speed, and accuracy. In addition to the default 4-bit conversion, you will find both uniform and mixed quantized files at various bit widths (2-bit, 3-bit, 6-bit, and 8-bit). This multi-quantized approach allows users to select the best variant for their deployment scenario, balancing precision and performance.

## Quantization Configurations

The model conversion uses a range of quantization configurations defined via `mlx_lm.convert`. These configurations fall into three main categories:

1. **Uniform Quantization:**
   Applies the same bit width to all layers.
   - **3bit:** Uniform 3-bit quantization.
   - **4bit:** Uniform 4-bit quantization (default).
   - **6bit:** Uniform 6-bit quantization.
   - **8bit:** Uniform 8-bit quantization.

2. **Mixed Quantization:**
   Uses a custom predicate function to decide the bit width for each layer—allowing different layers to use different precisions.
   - **2,6_mixed:** Uses the `mixed_2_6` predicate to choose between 2-bit and 6-bit quantization.
   - **3,6_mixed:** Uses the `mixed_3_6` predicate to choose between 3-bit and 6-bit quantization.
   - **3,4_mixed:** Built via `mixed_quant_predicate_builder(3, 4, group_size)`, it mixes 3-bit and 4-bit precision.
   - **4,6_mixed:** Built via `mixed_quant_predicate_builder(4, 6, group_size)`, it mixes 4-bit and 6-bit precision.
   - **4,8_mixed:** Built via `mixed_quant_predicate_builder(4, 8, group_size)`, it mixes 4-bit and 8-bit precision.

   Where `group_size = 64` (which is default for other quantization methods).

3. **Non-Quantized Conversions:**
   Converts the model to a different floating point precision without quantizing weights.
   - **bfloat16:** Model converted to bfloat16 precision.
   - **float16:** Model converted to float16 precision.

## Use with mlx

Install the MLX library:
```bash
pip install mlx-lm
```

Load the model and generate text:
```python
from mlx_lm import load, generate

model, tokenizer = load("mlx-community/DeepSeek-R1-Distill-Qwen-32B-MLX")

prompt = "hello"

if tokenizer.chat_template is not None:
    messages = [{"role": "user", "content": prompt}]
    prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)

response = generate(model, tokenizer, prompt=prompt, verbose=True)
```

Each configuration is optimized to meet specific requirements, enabling a forward-thinking approach in model deployment where resource constraints and performance targets are key considerations.