🦉 GitHub • 🤗 Model • 📚 Dataset Files
**Xing4.0-29B-Intent-oss** is the companion dataset for [Xing4.0-29B-Intent](https://huggingface.co/XingChen-AGI/Xing4.0-29B-Intent), a model for schema-guided intent recognition and structured information extraction. It is curated and released by the Xingchen General Artificial Intelligence Laboratory of China Telecom Artificial Intelligence Technology Co., Ltd. For complete schema definitions, conversion rules, and training/inference example assembly instructions, see the dataset documentation on GitHub: [English](https://github.com/XingChen-AGI/Xing4.0-29B-Intent/blob/main/data/README.md) | [简体中文](https://github.com/XingChen-AGI/Xing4.0-29B-Intent/blob/main/data/README_zh.md). ## Overview The dataset combines two sources: - **General**: 61 public information extraction and spoken language understanding datasets reconstructed into a unified schema-guided format. - **XingChen**: China Telecom's in-house intent and slot annotations for smart-device assistant scenarios. It supports supervised fine-tuning, schema-guided semantic understanding evaluation, and research on generalization across tasks, domains, and languages. Reproducing a specific evaluation result also requires matching data versions, evaluation scripts, and inference settings. ## Data Distribution | Statistic | Value | | --- | --- | | Total samples | 2,753,000 | | Reconstructed public datasets | 61 | | Domains | 19 | | Languages | 52, including Chinese and English | | Task types | NER, RE, EE, and intent recognition / slot filling | | Task | Samples | Share | Representative datasets | | --- | ---: | ---: | --- | | Named entity recognition (NER) | 823,000 | 29.9% | MSRA, OntoNotes, CoNLL2003, CMeEE, WeiboNER | | Relation extraction (RE) | 905,000 | 32.9% | DuIE, NYT, TACRED, BioRED, SemEval | | Event extraction (EE) | 56,000 | 2.0% | ACE05, RAMS, FewFC, MAVEN, DuEE | | Intent recognition / slot filling (SLU) | 969,000 | 35.2% | Snips, ATIS, MASSIVE | | **Total** | **2,753,000** | **100%** | — | ## Data Format Dataset files are JSON arrays, with one sample per element. Each sample includes its input conversation, a schema reference, and a serialized reference output. | Fields | Description | | --- | --- | | `id`, `turn` | Sample and conversation-turn identifiers | | `task`, `source`, `domain`, `language` | Task and provenance metadata | | `schema_path`, `schema` | Schema file path or an embedded schema array; a non-empty embedded array takes precedence | | `prompt_path`, `prompt` | Optional prompt file or embedded prompt template | | `dialogs` | Input turns containing `role` and `content` | | `output` | Reference answer stored as a JSON string | Illustrative sample: ```json { "id": "idx", "turn": "idx-0", "task": "Intent Recognition", "source": "XingChen", "domain": "Phone", "language": "Chinese", "schema_path": "./schema_description/schema.json", "schema": "", "prompt_path": "", "prompt": "", "dialogs": [ {"role": "user", "content": "明天的紫外线强度"} ], "output": "[{\"category\": \"查询天气\", \"args\": [{\"时间\": \"明天\", \"天气指数\": \"紫外线指数\"}]}]" } ``` The input schema defines candidate categories and their argument names and descriptions. The parsed output uses a common structure across all tasks: ```json [ { "category": "查询天气", "args": [{"时间": "明天", "天气指数": "紫外线指数"}] } ] ``` Here, `category` identifies the intent, entity type, relation type, or event type. Each object in `args` contains one set of extracted arguments. Multiple categories use separate top-level objects; multiple instances of a category use separate objects in `args`. Use `[]` when there are no arguments to fill. The `output` field must be parsed with `json.loads(output)` to obtain this structure. Schema and prompt paths are resolved relative to the corresponding data root, such as `General/test/` or `XingChen/`, rather than the individual JSON file's directory. ## Download and Use Browse and download the published files from the [Files and versions tab](https://huggingface.co/datasets/XingChen-AGI/Xing4.0-29B-Intent-oss/tree/main). Keep dataset files together with their associated schema and prompt files, preserving their relative paths. The GitHub repository provides [build_examples.py](https://github.com/XingChen-AGI/Xing4.0-29B-Intent/blob/main/data/build_examples.py) to assemble records into `messages` and `output` pairs. Follow the [setup and usage examples](https://github.com/XingChen-AGI/Xing4.0-29B-Intent/blob/main/data/README.md#usage-examples) to arrange the downloaded data and run the script from the model repository root. - **Training**: convert the assembled records to the format required by your training framework, for example by appending `output` as an assistant reply. - **Inference**: use `messages` as the request's message field and add parameters such as `model`. Never include reference answers in inference requests. - **Evaluation**: preserve task-specific single-label or multi-label constraints, construct each sample independently, and compare parsed output structures rather than raw JSON strings. ## License This dataset is released under the [Apache License 2.0](https://github.com/XingChen-AGI/Xing4.0-29B-Intent/blob/main/LICENSE). The original license terms of the public source datasets also apply; check their respective terms before commercial use.