LingLan Traditional Chinese Medicine
Benchmark Dataset
5 evaluation domains · 13 benchmark tasks · 25,620 Full items · 5,200 Hard items
LingLan is an expert-curated benchmark for evaluating large language models on Traditional Chinese Medicine knowledge and clinical reasoning across five domains and 13 benchmark tasks. This repository distributes the labelled data; evaluation code is maintained in the [GitHub project](https://github.com/TCMAI-BJTU/LingLan). The counts above treat each joint diagnostic and therapeutic reasoning case as one item. Its labels are stored in three files, giving **15 task configurations and 30 JSONL files**, with **29,620 Full and 6,000 Hard file rows**. Every record has `id`, `task`, and `answer`. **Hard is a subset of Full, not an independent train/test partition.** Each Hard file contains 400 records with the same IDs and labels as Full. Do not concatenate the two sets as independent examples. ## Load from ModelScope Install the dependencies: ```bash python -m pip install "modelscope[datasets]" ``` Load one task and one set: ```python from modelscope.msdatasets import MsDataset full = MsDataset.load("TCMAIBJTU/LingLan", subset_name="fundamental_tcm_knowledge_multiple_choice", split="fullset") hard = MsDataset.load("TCMAIBJTU/LingLan", subset_name="fundamental_tcm_knowledge_multiple_choice", split="hardset") print(len(full), len(hard)) # 1919, 400 print(hard[0]["id"], hard[0]["answer"]) ``` The README YAML defines the task configurations and their Full/Hard file mappings for the general dataset loader. The same mappings are available in `task_configs.json`. Choose a configuration from the table below. `fullset` and `hardset` are the split names. Always specify the task; different tasks have different answer schemas. ## Tasks and counts | Configuration / `task` | Full | Hard | | --- | ---: | ---: | | `chinese_patent_medicine_knowledge_single_choice` | 1,998 | 400 | | `chinese_patent_medicine_knowledge_fill_blank` | 2,000 | 400 | | `chinese_patent_medicine_knowledge_multiple_choice` | 1,950 | 400 | | `herbal_prescription_single_choice` | 2,000 | 400 | | `treatment_method_single_choice` | 2,000 | 400 | | `syndrome_single_choice` | 2,000 | 400 | | `herbal_prescription` | 2,000 | 400 | | `treatment_method` | 2,000 | 400 | | `syndrome` | 2,000 | 400 | | `fundamental_tcm_knowledge_single_choice` | 1,962 | 400 | | `fundamental_tcm_knowledge_fill_blank` | 1,959 | 400 | | `fundamental_tcm_knowledge_multiple_choice` | 1,919 | 400 | | `classical_text_entity_extraction` | 2,000 | 400 | | `clinical_record_entity_extraction` | 2,000 | 400 | | `tcm_licensing_examination` | 1,832 | 400 | ## Structure and fields ```text LingLan/ ├── README.md ├── README_zh.md ├── task_configs.json ├── data_manifest.json └── data/ ├── README.md ├── README_zh.md ├── fullset/ └── hardset/ ``` See [data/README.md](data/README.md) for all fields, answer formats, ID relationships, and evaluation output conventions. `task_configs.json` mirrors the README's task/file mappings. `data_manifest.json` records the row count and SHA-256 checksum of each JSONL file. The Hugging Face and ModelScope distributions contain identical JSONL bytes copied from the GitHub source data. ## Independent and joint diagnostic and therapeutic reasoning evaluation Diagnostic and therapeutic reasoning supports generating syndrome, treatment, or prescription independently, or all three in one call. Join the three files by four-digit case ID for joint generation; the GitHub runner uses this workflow. See [diagnostic and therapeutic reasoning documentation](data/README.md#diagnostic-and-therapeutic-reasoning) for fields, entry-point names, and scoring. This example loads the three gold answers for one shared case: ```python from modelscope.msdatasets import MsDataset tasks = ["syndrome", "treatment_method", "herbal_prescription"] components = {} for task in tasks: rows = MsDataset.load("TCMAIBJTU/LingLan", subset_name=task, split="hardset") components[task] = {r["id"].rsplit("_", 1)[1]: r for r in rows} case_id = next(iter(components["syndrome"])) records = [components[task][case_id] for task in tasks] assert len({r["病历文本"] for r in records}) == 1 print(records[0]["病历文本"]) print({task: components[task][case_id]["answer"] for task in tasks}) ``` ## Data sources and license Data are curated from de-identified clinical records, classical texts, official examination sources, and structured knowledge resources. The LingLan dataset is released under the [Creative Commons Attribution 4.0 International license](https://creativecommons.org/licenses/by/4.0/). Please credit [LingLan, its authors, and the data source](https://github.com/TCMAI-BJTU/LingLan), and cite the [paper](https://arxiv.org/abs/2602.01779) when using it. When sharing the data, retain the license link and indicate any changes. See [data/LICENSE](data/LICENSE) for the full terms. ## Citation ```bibtex @misc{hua2026linglan, title={LingLan: A Unified Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine}, author={Rui Hua and Yu Wei and Zixin Shu and Kai Chang and Dengying Yan and Jianan Xia and Zeyu Liu and Hui Zhu and Shujie Song and Mingzhong Xiao and Xiaodong Li and Dongmei Jia and Zhuye Gao and Yanyan Meng and Naixuan Zhao and Yu Fu and Haibin Yu and Benman Yu and Yuanyuan Chen and Fei Dong and Zhizhou Meng and Pengcheng Yang and Songxue Zhao and Lijuan Pei and Yunhui Hu and Kan Ding and Jiayuan Duan and Wenmao Yin and Yang Gu and Runshun Zhang and Qiang Zhu and Jian Yu and Jiansheng Li and Baoyan Liu and Wenjia Wang and Xuezhong Zhou}, year={2026}, eprint={2602.01779}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2602.01779}, } ```