---
title: nanochat-d34
canonical_url: "https://www.modelscope.cn/models/AI-ModelScope/nanochat-d34"
md_url: "https://www.modelscope.cn/models/AI-ModelScope/nanochat-d34.md"
repository: AI-ModelScope/nanochat-d34
last_updated: 2025-11-20
license: mit
library_name:
  - pytorch
frameworks:
  - pytorch
downloads: 54
stars: 0
---

# nanochat-d34

> nanochat-d34 - AI-ModelScope 在 ModelScope 开源的模型。nanochat-d34 model. It was pretrained like this:

AI-ModelScope/nanochat-d34 是 ModelScope 魔搭社区上的机器学习模型，采用 mit 许可。

- **Repository**: AI-ModelScope/nanochat-d34
- **License**: mit
- **Downloads**: 54
- **Stars**: 0
- **Last updated**: 2025-11-20

Source: https://www.modelscope.cn/models/AI-ModelScope/nanochat-d34

---

nanochat-d34 model. It was pretrained like this:

```bash
torchrun --standalone --nproc_per_node=8 -m scripts.base_train -- --depth=34 --device_batch_size=4 --target_param_data_ratio=40 --save_every=5000 --run=d34
```

On an 8XH100 node, which ran for \~100 hours (\~4 days) and cost \~$2,500. Notably, note that it is 2X longtrained compared to Chinchilla, i.e. the param:token ratio is overridden from 20 up to 40. This means the model was trained longer that is compute optimal, just to squeeze a bit more capability into a bit smaller package.

Some of the notable stats of the model are as follows:

```
- depth: 34
- max_seq_len: 2048
- target_param_data_ratio: 40
- device_batch_size: 4
- total_batch_size: 524,288
- Number of parameters: 2,217,082,880
- Number of FLOPs per token: 1.426509e+10
- Calculated number of iterations: 169,150
- Number of training tokens: 88,683,315,200
- Tokens : Params ratio: 40.0000
- DDP world size: 8
- Minimum validation bpb: 0.7045
- Final validation bpb: 0.7045
- MFU %: 47.40%
- Total training flops: 1.265075e+21
- Peak memory usage: 69811.85MiB
```

The achieved **CORE score** of this base model is **0.3382**. (For comparison, d32 is 0.3168, d20 "speedrun" is 0.22, GPT-2 is 0.25).

This upload allows you to skip base model pretraining and focus on finetuning, which is a lot cheaper, and saves you $2,500 of cost. To incorporate it into your nanochat repo it's a bit janky right now, basically download these files and then:

- the `token_bytes.pt`, `tokenizer.pkl` have to go into ~/.cache/nanochat/tokenizer directory
- the `meta_169150.json` and `model_169150.pt` have to go into ~/.cache/nanochat/chatsft_checkpoints/d34/
- 
I'll figure out how to make this less janky in the future, and to make nanochat play nicer with huggingface infra.