---
title: CCI3-HQ-Intermediate-Checkpoints
canonical_url: "https://www.modelscope.cn/models/BAAI/CCI3-HQ-Intermediate-Checkpoints"
md_url: "https://www.modelscope.cn/models/BAAI/CCI3-HQ-Intermediate-Checkpoints.md"
repository: BAAI/CCI3-HQ-Intermediate-Checkpoints
last_updated: 2024-10-29
parameters: 18.9B
tensor_type:
  - BF16
library_name:
  - safetensors
  - pytorch
frameworks:
  - pytorch
downloads: 1817
stars: 0
---

# CCI3-HQ-Intermediate-Checkpoints

> CCI3-HQ-Intermediate-Checkpoints - BAAI 在 ModelScope 开源的模型。CCI3-HQ-Intermediate-Checkpoints

- **Repository**: BAAI/CCI3-HQ-Intermediate-Checkpoints
- **Parameters**: 18.9B
- **Downloads**: 1817
- **Stars**: 0
- **Last updated**: 2024-10-29

Source: https://www.modelscope.cn/models/BAAI/CCI3-HQ-Intermediate-Checkpoints

---

# CCI3-HQ-Intermediate-Checkpoints

In the CCI3.0-HQ tech report, we conducted a direct comparison of different datasets through end-to-end pre-training experiments. The performance evaluation was based on both the final checkpoints and intermediate checkpoints of model training in two experiments: the **Mixed Dataset Experiment** and the **Chinese Dataset Experiment**. 

To closely monitor and compare the performance of various datasets throughout the training process, we saved intermediate checkpoints at approximately every 20 billion tokens of training. This allowed us to track progress and changes in performance over time.

Below, we list all checkpoints from the models trained in all comparison experiments. The suffix "-zh" represents checkpoints from the **Chinese Dataset Experiment**, while the suffix "-mix" indicates checkpoints from the **Mixed Dataset Experiment**.

## Citation Information
You can cite [our paper](https://arxiv.org/abs/2410.18505):

```
@misc{wang2024cci30hqlargescalechinesedataset,
      title={CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for pre-training large language models}, 
      author={Liangdong Wang and Bo-Wen Zhang and Chengwei Wu and Hanyu Zhao and Xiaofeng Shi and Shuhao Gu and Jijie Li and Quanyue Ma and TengFei Pan and Guang Liu},
      year={2024},
      eprint={2410.18505},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.18505}, 
}
```