---
title: dolma2-tokenizer
canonical_url: "https://www.modelscope.cn/models/LLM-Research/dolma2-tokenizer"
md_url: "https://www.modelscope.cn/models/LLM-Research/dolma2-tokenizer.md"
repository: LLM-Research/dolma2-tokenizer
last_updated: 2024-09-07
library_name:
  - pytorch
frameworks:
  - pytorch
downloads: 2135
stars: 0
---

# dolma2-tokenizer

> dolma2-tokenizer - LLM-Research 在 ModelScope 开源的模型。Slightly modified version of cl100kbase that supports Dolma 1.x special tokens (|||PHONENUMBER|||, |||EMAILADDRESS|||, |||IPADDRESS|||) as well as adds extra tokens to fill gaps in tiktoken cl100kbase version.

- **Repository**: LLM-Research/dolma2-tokenizer
- **Downloads**: 2135
- **Stars**: 0
- **Last updated**: 2024-09-07

Source: https://www.modelscope.cn/models/LLM-Research/dolma2-tokenizer

---

Slightly modified version of `cl100k_base` that supports Dolma 1.x special tokens 
(`|||PHONE_NUMBER|||`, `|||EMAIL_ADDRESS|||`, `|||IP_ADDRESS|||`) as well as adds 
extra tokens to fill gaps in tiktoken `cl100k_base` version.