---
title: MonkeyDoc
canonical_url: "https://www.modelscope.cn/datasets/zenosai/MonkeyDoc"
md_url: "https://www.modelscope.cn/datasets/zenosai/MonkeyDoc.md"
repository: zenosai/MonkeyDoc
chinese_name: MonkeyDoc
last_updated: 2026-01-30
license: "Apache License 2.0"
storage_size: "1.6 TB"
downloads: 112692
stars: 10
---

# MonkeyDoc

> MonkeyDoc - zenosai 在 ModelScope 开源的数据集。文档解析数据集MonkeyDoc

zenosai/MonkeyDoc 是 ModelScope 魔搭社区上的数据集，存储大小 1.6 TB，采用 Apache License 2.0 许可。

- **Repository**: zenosai/MonkeyDoc
- **License**: Apache License 2.0
- **Storage size**: 1.6 TB
- **Downloads**: 112692
- **Stars**: 10
- **Last updated**: 2026-01-30

Source: https://www.modelscope.cn/datasets/zenosai/MonkeyDoc

---

## Download and Extract

### 1) Download the dataset
Download the dataset to a local directory, for example:
````bash
modelscope download --repo-type dataset zenosai/MonkeyDoc --local-dir /path/to/MonkeyDoc
````

### 2) Extract all image archives

````bash
cd /path/to/MonkeyDoc

# Extract all .tar.zst archives under images into the images folder
for f in ./images/*.tar.zst; do
  tar -I zstd -xf "$f" -C .
done
````

### 3) (Optional) Add prefix to all image paths in MonkeyDoc.json
If needed, add the prefix `/path/to/MonkeyDoc/` to all image paths in `MonkeyDoc.json`.

````bash
python add_prefix.py \
  --input /path/to/MonkeyDoc/MonkeyDoc.json \
  --output /path/to/MonkeyDoc/MonkeyDoc_prefixed.json \
  --prefix /path/to/MonkeyDoc/
````

<details>
<summary><span style="font-size: 1.4em;"><b>Necessary details of the MonkeyDoc data generation pipeline</b></span></summary>


### 1. Text, Formula, and Table Recognition Data Generated by Qwen2.5VL-72B and Gemini-2.5-Pro

We use Qwen2.5-VL-72B to generate text and formula data. Since its table recognition performance does not meet our requirements, we rely exclusively on Gemini-2.5 Pro for table data generation. Here, we present the prompts used for recognition.

#### Text & Formula Recognition
**Model:** Qwen2.5VL-72B

```python
# Text Recognition Prompt
text_prompt = (
    "Please output the text content from the image. Inline formulas should be wrapped with $ "
    "and fully retain the format of the mathematical formulas. Do not wrap non-mathematical "
    "content with $. Avoid line breaks within sentences, but allow line breaks after complete "
    "sentences when appropriate for text structure (e.g., paragraph separation, list items, "
    "or logical section breaks)."
)

# Formula Recognition Prompt
formula_prompt = (
    "Please write out the expression of the formula in the image using LaTeX format. "
    "If there is a corresponding number for the formula, generate it as well. "
    "Do not output content unrelated to the formula in the image."
)
```

#### Table Recognition
**Model:** Gemini 2.5 Pro

```python
# Table Recognition Prompt
table_prompt = (
    "你是HTML编码助手，只输出纯<table><tr><td><th>代码，完全反映表格图像结构，"
    "无<style><script>、无CSS、无注释、无解释、无<html>等，只要紧凑表格代码，尽量少换行和空格。"
)
```

---

### 2. Crawling and Cleaning Data from arXiv

To further increase the diversity of our dataset, we also crawl data from arXiv. Specifically, we collect LaTeX source files from papers across disciplines such as physics, mathematics, and computer science, and extract tables and formulas from them. Since the tables are written in LaTeX and often contain unnecessary comments and redundant content, we first use Qwen2.5-72B to clean the LaTeX. The cleaned LaTeX is then converted into HTML using Qwen2.5-72B and finally rendered into table images. Here, we release the prompts used for cleaning as well as the CSS styles used for rendering.




#### Filtering & Conversion Prompts
**Model:** Qwen2.5-72B
###### LaTeX Code Filtering
Prompt to clean up raw LaTeX tables.

```python
filter_prompt = (
    "我希望将以下 LaTeX 表格渲染为图像，用于构建表格图文对，以训练多模态大模型。请你按照以下要求清洗表格代码：\n"
    "1. 保留表格的完整可视化结构和内容，包括：\n"
    " （1）表头、单元格数据、合并单元格；\n"
    " （2）所有用于边框绘制的命令，如 \\hline、\\cline、\\multicolumn 等；\n"
    " （3）所有在 tabular 环境内部的可视内容（包括中文、英文、符号、公式等）。\n"
    "2. 删除所有不会影响渲染效果的冗余代码，包括但不限于：\n"
    " （1）表格外围结构：\\label{}、\\caption{}、\\centering、\\resizebox{}、\\begin{table}、\\end{table}；\n"
    " （2）引用指令：\\cite{}、\\citet{}、\\citep{} 等；\n"
    " （3）行高设置指令：\\setlength\\extrarowheight{...}；\n"
    " （4）间距与位移控制命令：如 \\hspace{...}、\\vspace{...}、\\hskip ...、\\vskip ...、\\kern ...、\\rule{0pt}{...}；\n"
    " （5）所有类似 10em、5pt、1cm、2mm 等单位控制参数；\n"
    "3. 保留 tabular 环境和其内部结构完整性，包括 \\begin{tabular}、\\hline、\\multicolumn、\\cline、\\end{tabular} 等；\n"
    "4. 输出一个结构清晰、干净、无冗余、可直接渲染为图像的 LaTeX 表格代码，不要更改任何影响视觉结构的内容。\n"
    "5.对任何非标准 LaTeX 命令（例如自定义命令或拼写错误命令，如 \\mycmd, \\textbfy, \\specialcolor），"
    "如果可以替换为标准命令，则替换为标准 LaTeX 表格命令（例如 \\mrm{} → \\mathrm{}，\\bf{} → \\textbf{}，"
    "\\it{} → \\textit{}，\\tt{} → \\texttt{}），否则，直接删除该命令及其参数，仅保留命令中的文本内容"
    "（例如 \\Elow → Elow, \\Eup → Eup）；删除所有 \\newcommand, \\renewcommand, \\def 命令定义及其引用；"
    "所有看似命令但 LaTeX 无定义的专业符号（如科学单位、天文符号等），删除命令本身，仅保留其语义内容"
    "（如保留 'M$_\\odot$' 代替 \\msun）。"
    "以下是原始 LaTeX 表格代码，请按上述要求清洗，仅仅输出清洗后的latex代码，不要输出其他内容："
)
```

###### LaTeX to HTML Conversion
Prompt to convert clean LaTeX tables into HTML format.

```python
conversion_prompt = """你是一个擅长排版和格式转换的助手。我将提供一段包含表格的 LaTeX 代码，请你将其准确转换为标准 HTML 表格代码（使用 <table> 结构），并严格按照以下要求输出：

1.HTML 代码必须包含 <thead> 和 <tbody> 标签，区分表头和表体；
2.保留 LaTeX 中的结构信息，包括：
（1）单元格合并（如 \\multicolumn 和 \\multirow）；
（2）水平和垂直对齐方式（如 c、l、r）；
（3）单元格边框；
（4）文本样式（如 粗体 \\textbf{}、斜体 \\textit{}）；
3.所有数学公式应使用 $...$ 包围，并对上下标使用 {} 包围（例如 $x_{1}$, $x^{2}$ 等），保持 LaTeX 数学格式；
4.不要输出任何 CSS 样式（如 style="..."），也不要包含 <style> 标签或类名；
5.只返回最终的 HTML 代码，不要添加任何额外说明、解释或前后缀内容。
LaTeX 表格如下："""
```

#### CSS Styles for Rendering
Below is a collection of CSS styles used to render HTML tables into diverse visual styles for training our model.

```python
css_styles = [
    {
        "name": "Classic Blue",
        "css": """
        table { font-family: Arial, sans-serif; border-collapse: collapse; width: 80%; margin: 20px auto; box-shadow: 0 2px 5px rgba(0,0,0,0.1); }
        caption { font-size: 1.2em; margin: 0.5em 0; font-weight: bold; color: #333; }
        td, th { border: 1px solid #ddd; padding: 10px; text-align: left; }
        th { background-color: #4A90E2; color: white; font-weight: bold; }
        tr:nth-child(even) { background-color: #f2f8ff; }
        tr:hover { background-color: #d0e0f0; }
        """
    },
    {
        "name": "Modern Green",
        "css": """
        table { font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif; border-collapse: collapse; width: 90%; margin: 25px auto; border-radius: 8px; overflow: hidden; box-shadow: 0 4px 8px rgba(0,0,0,0.15); }
        caption { font-size: 1.3em; margin: 0.8em 0; font-weight: bold; color: #2E7D32; }
        td, th { padding: 12px 15px; text-align: left; }
        th { background-color: #4CAF50; color: white; }
        tbody tr { border-bottom: 1px solid #e0e0e0; }
        tbody tr:nth-child(odd) { background-color: #e8f5e9; }
        tbody tr:last-child { border-bottom: none; }
        tbody tr:hover { background-color: #c8e6c9; }
        """
    },
    {
        "name": "Minimalist Lines",
        "css": """
        table { font-family: 'Open Sans', sans-serif; border-collapse: collapse; width: 100%; margin: 20px 0; border-top: 2px solid #333; border-bottom: 2px solid #333; }
        caption { font-size: 1.1em; margin: 0.5em 0; font-weight: normal; color: #555; text-align: left; }
        td, th { padding: 10px 8px; text-align: left; border-bottom: 1px solid #eee; }
        th { color: #333; font-weight: bold; background-color: transparent; }
        tr:hover td { background-color: #f9f9f9; }
        """
    },
    {
        "name": "Orange Accent",
        "css": """
        table { font-family: Verdana, Geneva, sans-serif; border-collapse: separate; border-spacing: 0; width: 75%; margin: 20px auto; border: 1px solid #FF9800; border-radius: 6px; box-shadow: 0 1px 3px rgba(0,0,0,0.1); }
        caption { font-size: 1.25em; margin: 0.7em 0; font-weight: bold; color: #E65100; }
        td, th { border: 1px solid #FFCC80; padding: 9px; text-align: center; }
        th { background-color: #FF9800; color: white; }
        th:first-child { border-top-left-radius: 5px; } th:last-child { border-top-right-radius: 5px; }
        tr:last-child td:first-child { border-bottom-left-radius: 5px; } tr:last-child td:last-child { border-bottom-right-radius: 5px; }
        tbody tr:nth-child(even) { background-color: #FFF3E0; }
        tbody tr:hover { background-color: #FFE0B2; }
        """
    },
    {
        "name": "Purple Bliss",
        "css": """
        table { font-family: 'Lato', sans-serif; border-collapse: collapse; width: 88%; margin: 22px auto; box-shadow: 0 0 15px rgba(100, 50, 150, 0.2); border: 1px solid #9C27B0; }
        caption { font-size: 1.3em; margin: 0.7em 0; font-weight: bold; color: #7B1FA2; padding-bottom: 5px; border-bottom: 2px solid #CE93D8; }
        td, th { border: 1px dashed #E1BEE7; padding: 12px; text-align: left; }
        th { background-color: #9C27B0; color: white; text-transform: uppercase; letter-spacing: 1px; }
        tr:nth-child(odd) { background-color: #F3E5F5; }
        tr:hover { background-color: #E1BEE7; color: #4A148C; }
        """
    },
    {
        "name": "Teal Professional",
        "css": """
        table { font-family: 'Roboto', sans-serif; border-collapse: collapse; width: 95%; margin: 20px auto; border-radius: 4px; overflow: hidden; }
        caption { font-size: 1.2em; margin: 0.6em 0; font-weight: 500; color: #00796B; text-align: center; }
        td, th { padding: 14px 10px; text-align: left; border-bottom: 1px solid #B2DFDB; }
        th { background-color: #009688; color: white; font-weight: 600; }
        tbody tr:nth-child(even) { background-color: #E0F2F1; }
        tbody tr:hover { background-color: #A7FFEB; }
        """
    },
    {
        "name": "Monochrome Gray",
        "css": """
        table { font-family: Consolas, monaco, monospace; border-collapse: collapse; width: 90%; margin: 20px auto; border: 1px solid #616161; }
        caption { font-size: 1.1em; margin: 1em 0; font-weight: normal; color: #212121; text-align: left; letter-spacing: 0.5px; }
        td, th { border: 1px solid #BDBDBD; padding: 8px; text-align: left; }
        th { background-color: #757575; color: white; font-weight: normal; }
        tr:nth-child(even) { background-color: #EEEEEE; }
        tr:hover { background-color: #E0E0E0; }
        """
    },
    {
        "name": "Sunny Yellow",
        "css": """
        table { font-family: 'Comic Sans MS', cursive, sans-serif; border-collapse: collapse; width: 82%; margin: 20px auto; border: 2px solid #FFC107; border-radius:10px; box-shadow: 0px 0px 10px #FFC107; }
        caption { font-size: 1.3em; margin: 0.7em 0; font-weight: bold; color: #FFA000; }
        td, th { border: 1px dotted #FFECB3; padding: 10px; text-align: center; }
        th { background-color: #FFC107; color: #424242; }
        tr:nth-child(odd) { background-color: #FFF9C4; }
        tr:hover { background-color: #FFF176; }
        """
    },
    {
        "name": "Deep Indigo",
        "css": """
        table { font-family: 'Georgia', serif; border-collapse: collapse; width: 85%; margin: 20px auto; background-color: #E8EAF6; color: #1A237E; border: 2px solid #3F51B5; }
        caption { font-size: 1.25em; margin: 0.8em 0; font-weight: bold; color: #303F9F; padding: 5px; background-color: #C5CAE9; }
        td, th { border: 1px solid #9FA8DA; padding: 11px; text-align: left; }
        th { background-color: #3F51B5; color: white; font-style: italic; }
        tr:hover { background-color: #9FA8DA; color: white; }
        """
    },
    {
        "name": "Forest Green",
        "css": """
        table { font-family: 'Trebuchet MS', Helvetica, sans-serif; border-collapse: collapse; width: 92%; margin: 20px auto; border-radius: 8px; overflow: hidden; border: 1px solid #1B5E20; }
        caption { font-size: 1.2em; margin: 0.6em 0; font-weight: bold; color: #2E7D32; }
        td, th { padding: 10px; text-align: left; border-right: 1px solid #A5D6A7; border-bottom: 1px solid #A5D6A7;}
        td:last-child, th:last-child { border-right: none; }
        th { background-color: #388E3C; color: #E8F5E9; }
        tbody tr:nth-child(even) { background-color: #C8E6C9; }
        tbody tr:hover { background-color: #81C784; }
        """
    },
    {
        "name": "Sky Blue Light",
        "css": """
        table { font-family: 'Verdana', sans-serif; border-collapse: collapse; width: 100%; margin: 15px 0; box-shadow: 0 1px 2px rgba(0,0,0,0.1); }
        caption { font-size: 1.1em; margin: 0.5em 0; font-weight: normal; color: #0277BD; text-align: center; padding-bottom: 8px; border-bottom: 1px solid #B3E5FC; }
        td, th { padding: 12px; text-align: left; border: none; }
        th { background-color: #03A9F4; color: white; font-weight: 500; }
        tbody tr { border-bottom: 1px solid #E1F5FE; }
        tbody tr:nth-child(odd) { background-color: #F4FCFF; }
        tbody tr:hover { background-color: #B3E5FC; }
        """
    },
    {
        "name": "Warm Brown",
        "css": """
        "name": "Warm Brown",
        "css": """
        table { font-family: 'Palatino Linotype', 'Book Antiqua', Palatino, serif; border-collapse: separate; border-spacing: 0; width: 80%; margin: 20px auto; border: 2px solid #5D4037; background-color: #EFEBE9; }
        caption { font-size: 1.3em; margin: 0.7em 0; font-weight: bold; color: #4E342E; }
        td, th { border: 1px solid #A1887F; padding: 9px; text-align: left; }
        th { background-color: #6D4C41; color: #D7CCC8; }
        tbody tr:nth-child(even) { background-color: #D7CCC8; }
        tbody tr:hover { background-color: #BCAAA4; color: #3E2723; }
        """
    },
    {
        "name": "Pink Delight",
        "css": """
        table { font-family: 'Brush Script MT', cursive; border-collapse: collapse; width: 78%; margin: 25px auto; border: 2px dashed #E91E63; border-radius: 15px; background-color: #FCE4EC; }
        caption { font-size: 1.5em; margin: 0.6em 0; font-weight: normal; color: #C2185B; }
        td, th { border: 1px solid #F8BBD0; padding: 12px; text-align: center; color: #880E4F; }
        th { background-color: #F06292; color: white; font-size: 1.1em; }
        tr:hover { background-color: #F48FB1; font-weight: bold; }
        """
    },
    {
        "name": "Graphite Minimal",
        "css": """
        table { font-family: 'Helvetica Neue', Arial, sans-serif; border-collapse: collapse; width: 95%; margin: 20px auto; border: none; }
        caption { font-size: 1.15em; margin: 1em 0 0.5em 0; font-weight: 300; color: #333; text-align: left; }
        td, th { padding: 12px 8px; text-align: left; border-bottom: 1px solid #ccc; }
        th { background-color: #f0f0f0; color: #333; font-weight: 600; border-top: 2px solid #555; border-bottom: 2px solid #555; }
        tbody tr:hover td { background-color: #e9e9e9; }
        """
    },
    {
        "name": "Autumn Hues",
        "css": """
        table { font-family: 'Garamond', serif; border-collapse: collapse; width: 90%; margin: 20px auto; border: 1px solid #BF360C; }
        caption { font-size: 1.3em; margin: 0.6em 0; font-weight: bold; color: #D84315; }
        td, th { border: 1px solid #FFCCBC; padding: 10px; text-align: left; }
        th { background-color: #EF6C00; color: #FFF3E0; }
        tr:nth-child(even) { background-color: #FF therapiesE0; } /* Corrected to #FFE0B2 or similar light orange/beige */
        tr:nth-child(odd) { background-color: #FFF9C4; }
        tr:hover { background-color: #FFAB91; color: #BF360C; }
        """
    },
    {
        "name": "Clean Slate",
        "css": """
        table { font-family: 'Calibri', sans-serif; border-collapse: collapse; width: 100%; margin: 15px 0; }
        caption { font-size: 1.2em; margin-bottom: 10px; font-weight: bold; color: #444; text-align: left; }
        td, th { padding: 10px 15px; text-align: left; border-bottom: 1px solid #e0e0e0; }
        th { background-color: #f7f7f7; color: #333; font-weight: bold; border-top: 1px solid #d0d0d0; }
        tbody tr:hover { background-color: #f0f0f0; }
        """
    },
    {
        "name": "Classic Three-Line",
        "css": """
        table {
            font-family: 'Times New Roman', Times, serif; 
            border-collapse: collapse; 
            width: 90%;
            margin: 25px auto;
            border-top: 2px solid black;    
            border-bottom: 2px solid black; 
        }
        caption {
            font-size: 1.2em;
            margin: 0.8em 0;
            font-weight: bold;
            color: #333;
            text-align: center; 
        }
        th {
            padding: 10px 12px;
            text-align: left;
            border-bottom: 1px solid black; 
            font-weight: bold;
            background-color: transparent; 
            color: black;
        }
        td {
            padding: 10px 12px;
            text-align: left;
            border: none; 
        }
        """
    },
    {
        "name": "Pure No-Line (Borderless)",
        "css": """
        table {
            font-family: 'Arial', sans-serif;
            border-collapse: collapse;
            width: 90%;
            margin: 20px auto;
            border: none; 
        }
        caption {
            font-size: 1.1em;
            margin: 0.6em 0;
            color: #444;
            text-align: left;
        }
        th {
            padding: 12px 10px;
            text-align: left;
            border: none; 
            font-weight: bold;
            background-color: #f8f8f8; 
            color: #333;
        }
        td {
            padding: 12px 10px;
            text-align: left;
            border: none; 
        }
        """
    },
    {
        "name": "Full-Line Grid",
        "css": """
        table {
            font-family: 'Verdana', Geneva, sans-serif;
            border-collapse: collapse; 
            width: 85%;
            margin: 20px auto;
            border: 1px solid #666; 
            box-shadow: 0 1px 3px rgba(0,0,0,0.1);
        }
        caption {
            font-size: 1.2em;
            margin: 0.7em 0;
            font-weight: bold;
            color: #333;
        }
        th {
            padding: 10px;
            text-align: left;
            border: 1px solid #999; 
            background-color: #e0e0e0; 
            color: #333;
            font-weight: bold;
        }
        td {
            padding: 10px;
            text-align: left;
            border: 1px solid #999; 
        }
        tr:nth-child(even) {
            background-color: #f9f9f9; 
        }
        tr:hover {
            background-color: #f1f1f1; 
        }
        """
    }
]
```


### 3. Synthetic Chinese Formula

Since most formulas in open-source datasets are in English, while Chinese formulas are common across various fields, we enrich our training data by designing prompts that enable large language models to automatically generate mathematical formulas for diverse industries.

First, we defined the industry categories and the corresponding domains of formulas.

```python
comprehensive_industry_formulas = [
    {"行业": "自然科学", "公式类别": ["基础代数/微积分", "微分方程", "统计与概率", "向量与矩阵运算"]},
    {"行业": "物理学", "公式类别": ["经典力学公式", "电磁学公式", "量子力学方程", "热力学定律", "相对论公式"]},
    {"行业": "化学", "公式类别": ["化学计量学公式", "反应动力学方程", "热力学函数表达式", "量子化学方程"]},
    {"行业": "生物学", "公式类别": ["种群动力学模型", "生物统计学公式", "遗传平衡定律", "酶动力学方程", "生物能学公式"]},
    {"行业": "天文学", "公式类别": ["天体力学方程", "恒星演化模型", "宇宙学红移公式", "星等计算模型"]},
    {"行业": "地质学", "公式类别": ["岩石力学公式", "放射性定年公式", "板块运动模型", "地质统计学方程"]},
    {"行业": "气象学/气候学", "公式类别": ["大气动力学方程", "辐射传输公式", "气候模型方程", "热力学湿空气公式"]},
    {"行业": "工程学", "公式类别": ["机械工程公式", "电气工程定律", "流体力学方程", "材料力学公式", "控制系统模型"]},
    {"行业": "计算机科学", "公式类别": ["算法复杂度分析", "密码学算法公式", "图形学变换矩阵", "机器学习损失函数", "信息论公式"]},
    {"行业": "人工智能/机器学习", "公式类别": ["神经网络反向传播公式", "优化算法迭代公式", "聚类分析距离公式", "自然语言处理概率模型"]},
    {"行业": "数据科学/大数据", "公式类别": ["数据清洗统计量公式", "特征工程变换公式", "分布式计算模型"]},
    {"行业": "数学本身", "公式类别": ["数论公式", "拓扑学方程", "复变函数公式", "泛函分析表达式", "分形几何模型"]},
    {"行业": "金融学", "公式类别": ["资产定价模型", "风险度量公式", "期权定价方程", "投资组合优化模型", "利率计算模型"]},
    {"行业": "经济学", "公式类别": ["供需模型方程", "宏观经济增长模型", "博弈论支付函数", "投入产出分析公式"]},
    {"行业": "会计学/审计", "公式类别": ["财务比率计算公式", "折旧计算模型", "现值/终值公式", "审计抽样统计公式"]},
    {"行业": "市场营销", "公式类别": ["市场份额计算公式", "客户生命周期价值（CLV）模型", "定价策略弹性公式"]},
    {"行业": "管理学", "公式类别": ["决策树模型概率公式", "关键路径法（CPM）时间计算", "库存管理EOQ模型"]},
    {"行业": "教育学", "公式类别": ["教育统计量公式", "学习曲线模型", "教育评价指数公式"]},
    {"行业": "心理学", "公式类别": ["心理测量信效度公式", "回归分析预测模型", "认知科学计算模型"]},
    {"行业": "社会学", "公式类别": ["社会网络分析中心性指标", "人口结构预测模型", "基尼系数计算公式"]},
    {"行业": "医学/生物医学工程", "公式类别": ["生理指标计算公式", "药物代谢动力学模型", "医学影像重建算法", "流行病学统计公式"]},
    {"行业": "药学", "公式类别": ["药物剂量计算模型", "药物相互作用概率公式", "制剂稳定性方程"]},
    {"行业": "护理学", "公式类别": ["液体平衡计算公式", "营养需求能量方程", "护理工作量评估模型"]},
    {"行业": "农业科学", "公式类别": ["作物生长模型方程", "农药残留降解公式", "农业生态效率计算", "畜牧养殖量预测模型"]},
    {"行业": "环境科学", "公式类别": ["污染物扩散模型", "生态承载力计算公式", "碳循环动力学方程", "环境经济核算公式"]},
    {"行业": "建筑学/土木工程", "公式类别": ["结构力学公式", "材料强度计算模型", "建筑热工设计公式", "施工进度计划模型"]},
    {"行业": "航空航天工程", "公式类别": ["空气动力学方程", "火箭推进公式", "轨道力学模型", "航天器姿态控制方程"]},
    {"行业": "交通运输工程", "公式类别": ["交通流量模型", "道路设计几何公式", "物流路径优化算法", "车辆动力学方程"]},
    {"行业": "能源工程", "公式类别": ["热力学循环效率公式", "电力系统潮流计算", "能源需求预测模型", "可再生能源转换效率公式"]},
    {"行业": "冶金工程", "公式类别": ["冶金反应平衡常数公式", "炉料配比计算模型", "金属凝固动力学方程"]},
    {"行业": "化学工程", "公式类别": ["化工热力学公式", "传质传热方程", "化学反应工程模型", "化工流程模拟算法"]},
    {"行业": "食品科学", "公式类别": ["食品成分分析统计公式", "食品保质期预测模型", "食品加工动力学方程"]},
    {"行业": "纺织工程", "公式类别": ["纤维力学性能计算公式", "织物结构参数模型", "纺织工艺优化方程"]},
    {"行业": "海洋科学", "公式类别": ["海浪动力学方程", "海洋环流模型", "海洋生态系统能量传递公式", "潮汐预测模型"]},
    {"行业": "水文水资源", "公式类别": ["水文统计公式", "径流计算模型", "地下水运动方程", "水资源供需平衡公式"]},
    {"行业": "采矿工程", "公式类别": ["矿石储量计算模型", "矿山压力分布公式", "采矿设备力学方程"]},
    {"行业": "安全工程", "公式类别": ["风险概率评估公式", "事故树分析（FTA）模型", "安全阈值计算方程"]},
    {"行业": "体育科学", "公式类别": ["运动生物力学公式", "运动训练负荷模型", "运动员体能指标计算公式"]},
    {"行业": "艺术/设计", "公式类别": ["黄金分割比例公式", "色彩空间转换矩阵", "视觉传达构图数学模型"]},
    {"行业": "音乐学", "公式类别": ["音阶频率计算模型", "和声理论数学规律", "乐器声学振动方程"]},
    {"行业": "考古学", "公式类别": ["碳14测年公式", "考古统计学断代模型", "文物修复几何还原算法"]},
    {"行业": "语言学", "公式类别": ["语言统计模型", "语法树生成概率公式", "语音声学特征参数计算"]},
    {"行业": "法学/刑侦", "公式类别": ["犯罪地理画像模型", "证据统计显著性检验公式", "法律经济分析模型"]},
    {"行业": "哲学/逻辑学", "公式类别": ["命题逻辑真值表公式", "模态逻辑表达式", "博弈论哲学模型"]},
    {"行业": "新闻学/传播学", "公式类别": ["传播效果量化模型", "舆情分析情感计算方程", "新闻传播统计公式"]},
    {"行业": "历史学", "公式类别": ["历史计量学统计模型", "人口史预测公式", "历史事件时间序列分析"]},
    {"行业": "宗教学", "公式类别": ["宗教人口分布数学模型", "宗教文本计量分析公式"]},
    {"行业": "区块链/加密货币", "公式类别": ["哈希函数算法公式", "共识机制概率模型", "通证经济学公式"]},
    {"行业": "游戏开发", "公式类别": ["物理引擎碰撞检测公式", "游戏经济系统平衡模型", "路径寻路算法"]},
    {"行业": "虚拟现实（VR）/增强现实（AR）", "公式类别": ["三维坐标变换矩阵", "传感器数据融合算法", "沉浸式体验数学模型"]},
    {"行业": "航天医学", "公式类别": ["失重环境生理指标模型", "航天员营养需求计算公式", "太空辐射剂量评估方程"]},
    {"行业": "灾害科学", "公式类别": ["地震震级计算模型", "洪水风险评估公式", "灾害损失统计方程"]},
    {"行业": "军事科学", "公式类别": ["弹道学轨迹方程", "军事运筹学模型", "装备效能评估公式"]},
    {"行业": "城市规划", "公式类别": ["人口密度计算公式", "土地利用优化模型", "城市交通流数学模型"]},
]
```

We have designed a dedicated system prompt for generating Chinese LaTeX formulas, ensuring that Chinese elements are correctly embedded in the formulas following LaTeX syntax rules.

```python
gen_cn_latex_system_prompt = """
【要求】
你作为LaTeX公式生成器，必须满足以下条件：  
1. 每个公式至少包含1个中文元素（变量/下标/条件/注释）  
2. 中文必须通过`\text{}`或`\mbox{}`实现，示例：  
   - 正确：`\sum_{\text{样本}=1}^N`  
   - 错误：`\sum_{样本=1}^N`  
3. 不需要解释, 输出单个公式，参考【强化示例库】

【中文应用场景】  
请根据需求将中文应用于以下位置：  
1. 等式两侧：`\text{动能} = \frac{1}{2}mv^2`  
2. 函数参数：`f(\text{温度}, \text{压强})`  
3. 中文变量：`f(x) = 收入 * t`  
4. 中文下标：`x_{\text{初始}}`
5. 条件范围：`\forall x \in \text{定义域}`  
6. 注释说明：`\underbrace{a+b}_{\text{基础项}}`  

【示例】  
输入：
方程组
输出：
\begin{cases}  
\text{电压} = I \times \text{阻抗} \\  
\text{功率} = \frac{\text{能量}}{\text{时间}}  
\end{cases}  

输入：
矩阵 
输出：
\text{变换矩阵} = \begin{pmatrix}  
\cos\theta & -\sin\theta \\  
\sin\theta & \cos\theta  
\end{pmatrix}_{\text{旋转}}  

输入：
极限表达式 
输出： 
\lim_{\text{误差} \to 0} \frac{\Delta \text{位移}}{\Delta t}  

输入：
导数应用 
输出：
\frac{d\text{成本}}{dt} = \text{边际成本} + \epsilon_{\text{波动项}}  """
```

We also designed a prompt to convert English formulas into Chinese ones.

```python
convert_cn_latex_system_prompt = """
【任务指令】
请将用户提供的纯英文LaTeX公式进行以下处理：
- 在保持数学语义不变的前提下，添加至少1个中文元素
- 中文必须通过\text{}或\mbox{}实现
- 只输出最终修改后的公式（不带任何解释）
- 不要输出分析过程和解释内容

【中文插入策略】
根据公式类型选择合适的中文应用场景：
- 方程/等式：添加中文注释或命名（如\text{总成本} = ...）
- 函数/参数：用中文描述参数意义（如f(\text{温度})）
- 变量替换：将英文变量替换为中文变量（如x → \text{输入}）
- 条件说明：用中文标注约束条件（如\forall x \in \text{有效范围}）
- 矩阵/向量：添加中文说明标签
- 导数/积分：添加中文注释说明

【操作规范】
- 保留原有公式结构和数学符号，但中文替换部分不保留
- 优先选择不会改变公式计算逻辑的位置插入中文
- 中文元素应与公式内容相关（如物理量/专业术语/过程说明）
- 使用标准LaTeX语法格式输出

【示例演示】
输入：
E = mc^2
输出:
\text{能量} = \text{质量} \times c^2

输入：
\frac{dy}{dx} = 2x
输出：
\frac{d\text{输出}}{dx} = 2x

输入：
\begin{cases}
y = ax + b \\
z = cx + d
\end{cases}
输出：
\begin{cases}
\text{预测值} = a\text{特征} + b \\
\text{误差} = c\text{特征} + d
\end{cases}
"""
```

Finally, we generate Chinese formulas based on the predefined prompts, industry categories, and the input English formulas. Below is a reference code for synthesizing Chinese formulas.

```python
import json
from tqdm import tqdm

# 注：client_infer为模型推理客户端函数，需根据实际使用的大模型接口实现
# 函数功能：接收输入文本和系统提示，返回模型生成的LaTeX公式；生成失败返回'error'
# def client_infer(input_text, system_prompt):
#     # 模型推理逻辑实现
#     pass

# 生成中文公式并保存
def generate_chinese_formulas(comprehensive_industry_formulas, gen_cn_prompt, out_file):
    with open(out_file, 'a+', encoding='utf8') as fw:
        for item in tqdm(comprehensive_industry_formulas):
            cat_list = [item['行业']]
            cat_list.extend(item['公式类别'])
            for input_text in cat_list:
                # 每个输入生成10个不同的公式，提升数据多样性
                for i in range(10):
                    # 调用模型推理接口生成中文LaTeX公式
                    res = client_infer(input_text, gen_cn_prompt)
                    if res == 'error':
                        continue  # 跳过生成失败的情况
                    out = {
                        'input': input_text, 
                        'latex_zh': res
                    }
                    fw.write(json.dumps(out, ensure_ascii=False) + '\n')

# 调用示例（需先实现client_infer函数）
# out_file = "chinese_formulas.jsonl"  # 输出文件路径
# generate_chinese_formulas(comprehensive_industry_formulas, gen_cn_latex_system_prompt, out_file)
```




### 4. Data Filtering

After data synthesis, we randomly sample a large number of instances for manual inspection to identify common error patterns. We then apply rule-based filtering to eliminate most data that are likely to contain errors. The filtering operations described below are for reference only and should be tailored to the specific dataset during actual data filtering.

```python
import json
from tqdm import tqdm
from PIL import Image
import re
import random
from bs4 import BeautifulSoup

def save_json(json_list, save_path):
    with open(save_path, 'w') as file:
        json.dump(json_list, file, indent=4, ensure_ascii=False)

def find_unescaped_symbols_in_html(html):
    """
    Find formula symbols in HTML tables that are NOT wrapped in '$'.
    :param html: HTML string containing the table
    :return: List of problem items
    """
    formula_symbol_regex = re.compile(r'[\^_\\{}]')
    soup = BeautifulSoup(html, 'lxml')
    problems = []
    
    cells = soup.find_all(['td', 'th'])
    for cell in cells:
        text = cell.get_text()
        parts = text.split('$')
        for i in range(0, len(parts), 2):
            outside_text = parts[i]
            if formula_symbol_regex.search(outside_text):
                problems.append({
                    "cell_text": text,
                    "found_in": outside_text.strip()
                })
                break 
    return problems

def has_ldots_outside_math(s):
    parts = re.split(r'(\$.*?\$)', s, flags=re.DOTALL)
    for i, part in enumerate(parts):
        if i % 2 == 0 and r'\ldots' in part: return True
    return False

def has_vdots_outside_math(s):
    parts = re.split(r'(\$.*?\$)', s, flags=re.DOTALL)
    for i, part in enumerate(parts):
        if i % 2 == 0 and r'\vdots' in part: return True
    return False

def has_verb_outside_math(s):
    parts = re.split(r'(\$.*?\$)', s, flags=re.DOTALL)
    for i, part in enumerate(parts):
        if i % 2 == 0 and r'\verb' in part: return True
    return False

def has_100_outside_math(s):
    parts = re.split(r'(\$.*?\$)', s, flags=re.DOTALL)
    for i, part in enumerate(parts):
        if i % 2 == 0 and r'\%' in part: return True
    return False

def has_and_outside_math(s):
    parts = re.split(r'(\$.*?\$)', s, flags=re.DOTALL)
    for i, part in enumerate(parts):
        if i % 2 == 0 and r'\&' in part: return True
    return False

def has_html_inside_dollar_math(s):
    pattern = re.compile(r'\$(.*?)\$', re.DOTALL)
    for match in pattern.finditer(s):
        content_inside = match.group(1)
        if re.search(r'<[^>]+>', content_inside): return True
    return False

def has_textsubscript_outside_math(s):
    parts = re.split(r'(\$.*?\$)', s, flags=re.DOTALL)
    for i, part in enumerate(parts):
        if i % 2 == 0 and r'\textsubscript' in part: return True
    return False

# --- Main Execution Block ---

json_paths = [""]

if __name__=="__main__":
    for json_path in json_paths:
        if not json_path: continue 
        
        with open(json_path,"r") as f:
            data = json.load(f)
        new_data = []
        
        for i in tqdm(range(len(data))):
            html = data[i]['messages'][-1]['content']
            
            # Filtering Rules
            if "\\textbf" in html: continue
            if "\\textbackslash" in html: continue
            if "\\ $n$" in html: continue
            if "\\textsc{i}" in html: continue
            if "\\AA{}" in html: continue
            if has_vdots_outside_math(html): continue
            if has_ldots_outside_math(html): continue
            if has_verb_outside_math(html): continue
            if has_100_outside_math(html): continue
            if has_html_inside_dollar_math(html): continue
            if r"\bm" in html: continue
            if has_textsubscript_outside_math(html): continue
            if has_and_outside_math(html): continue
            if "\\cite" in html: continue
            if "\\ref" in html: continue
            if "$\\AA" in html: continue
            if "$\\{Q, \\Omega\\} \\gets$" in html: continue
            
            # Structure Checks
            if not html.endswith("</table>"): continue
            if not html.startswith("<table>"): continue

            new_data.append(data[i])
            
        print(f"Processed {json_path}: Original {len(data)} -> Filtered {len(new_data)}")
        save_json(new_data, json_path)
```
</details>
