---
title: qwen3-from-scratch-distill-checkpoints
canonical_url: "https://www.modelscope.cn/models/AI-ModelScope/qwen3-from-scratch-distill-checkpoints"
md_url: "https://www.modelscope.cn/models/AI-ModelScope/qwen3-from-scratch-distill-checkpoints.md"
repository: AI-ModelScope/qwen3-from-scratch-distill-checkpoints
last_updated: 2026-03-24
license: apache-2.0
library_name:
  - pytorch
frameworks:
  - pytorch
language:
  - en
downloads: 8
stars: 0
tags:
  - pytorch
  - qwen3
  - distillation
  - reasoning
  - math
---

# qwen3-from-scratch-distill-checkpoints

> qwen3-from-scratch-distill-checkpoints - AI-ModelScope 在 ModelScope 开源的模型。Qwen3 From Scratch Distillation Checkpoints

AI-ModelScope/qwen3-from-scratch-distill-checkpoints 是 ModelScope 魔搭社区上的机器学习模型，采用 apache-2.0 许可。

- **Repository**: AI-ModelScope/qwen3-from-scratch-distill-checkpoints
- **License**: apache-2.0
- **Tags**: pytorch, qwen3, distillation, reasoning, math
- **Downloads**: 8
- **Stars**: 0
- **Last updated**: 2026-03-24

Source: https://www.modelscope.cn/models/AI-ModelScope/qwen3-from-scratch-distill-checkpoints

---

# Qwen3 From Scratch Distillation Checkpoints

This repository contains chapter 8 distillation checkpoints for the [rasbt/qwen3-from-scratch](https://huggingface.co/rasbt/qwen3-from-scratch) model from [Build a Reasoning Model (From Scratch)](https://github.com/rasbt/reasoning-from-scratch).

These files are raw PyTorch `state_dict` checkpoints intended for use with the `reasoning_from_scratch` package.

 
## Available Checkpoints

- [ch08_distill_deepseek_r1](https://huggingface.co/rasbt/qwen3-from-scratch-distill-checkpoints/tree/main/ch08_distill_deepseek_r1): the 3 DeepSeek-R1 distillation checkpoints used for rows 3-5 in [`ch08_main.ipynb`](https://github.com/rasbt/reasoning-from-scratch/blob/main/ch08/01_main-chapter-code/ch08_main.ipynb)
- [ch08_distill_qwen3_235b_a22b](https://huggingface.co/rasbt/qwen3-from-scratch-distill-checkpoints/tree/main/ch08_distill_qwen3_235b_a22b): the 3 Qwen3 235B A22B distillation checkpoints used for rows 6-8 in [`ch08_main.ipynb`](https://github.com/rasbt/reasoning-from-scratch/blob/main/ch08/01_main-chapter-code/ch08_main.ipynb)

 
## Usage Example

The checkpoints in this repository are intended for use with the `reasoning_from_scratch` package.

For a DeepSeek-R1 distillation checkpoint, you can download it via:

```python
from reasoning_from_scratch.qwen3 import download_qwen3_distill_checkpoints

download_qwen3_distill_checkpoints(
    distill_type="deepseek_r1",
    step="06682",
    out_dir="qwen3",
)
```

For a Qwen3 235B A22B distillation checkpoint, use:

```python
from reasoning_from_scratch.qwen3 import download_qwen3_distill_checkpoints

download_qwen3_distill_checkpoints(
    distill_type="qwen3_235b_a22b",
    step="05746",
    out_dir="qwen3",
)
```

Once downloaded, you can load a checkpoint and stream text as follows:

```python
from pathlib import Path
import torch

from reasoning_from_scratch.ch02 import (
    get_device,
    generate_text_basic_stream_cache,
)
from reasoning_from_scratch.qwen3 import (
    download_qwen3_distill_checkpoints,
    download_qwen3_small,
    Qwen3Model,
    Qwen3Tokenizer,
    QWEN_CONFIG_06_B,
)

device = get_device()
local_dir = Path("qwen3")

checkpoint_path = download_qwen3_distill_checkpoints(
    distill_type="deepseek_r1",
    step="06682",
    out_dir=local_dir,
)
download_qwen3_small(kind="reasoning", tokenizer_only=True, out_dir=local_dir)

tokenizer = Qwen3Tokenizer(
    tokenizer_file_path=local_dir / "tokenizer-reasoning.json",
    apply_chat_template=True,
    add_generation_prompt=True,
    add_thinking=True,
)
model = Qwen3Model(QWEN_CONFIG_06_B)
state_dict = torch.load(checkpoint_path, map_location=device)
model.load_state_dict(state_dict)
model.to(device)
model.eval()

prompt = "Solve: If x + 7 = 19, what is x?"
input_ids = torch.tensor(tokenizer.encode(prompt), device=device).unsqueeze(0)

for token in generate_text_basic_stream_cache(
    model=model,
    token_ids=input_ids,
    max_new_tokens=256,
    eos_token_id=tokenizer.eos_token_id,
):
    token_id = token.squeeze(0).item()
    print(tokenizer.decode([token_id]), end="", flush=True)
print()
```

 
## Notes

- These are the exact epoch checkpoints used in the chapter 8 results table in [`ch08_main.ipynb`](https://github.com/rasbt/reasoning-from-scratch/blob/main/ch08/01_main-chapter-code/ch08_main.ipynb).
- Both checkpoint families should be used with the reasoning tokenizer for consistency with chapter 8.

 
## Download Helper Reference

These are the supported `distill_type` values for `download_qwen3_distill_checkpoints(...)`:

- DeepSeek-R1 distillation data:
```python
download_qwen3_distill_checkpoints(
    distill_type="deepseek_r1",
    step="06682",
    out_dir="qwen3",
)
```

Available DeepSeek-R1 saved steps: `06682`, `13364`, `20046`.

- Qwen3 235B A22B distillation data:
```python
download_qwen3_distill_checkpoints(
    distill_type="qwen3_235b_a22b",
    step="05746",
    out_dir="qwen3",
)
```

Available Qwen3 235B A22B saved steps: `05746`, `11492`, `17238`.