--- title: data-product-benchmark canonical_url: "https://www.modelscope.cn/datasets/ibm-research/data-product-benchmark" md_url: "https://www.modelscope.cn/datasets/ibm-research/data-product-benchmark.md" repository: ibm-research/data-product-benchmark last_updated: 2026-03-19 license: mit storage_size: "876 MB" downloads: 294 stars: 0 --- # data-product-benchmark > data-product-benchmark - ibm-research 在 ModelScope 开源的数据集。This dataset provides a benchmark for automatic data product creation. The task is framed as follows: given a natural language data product request and a corpus of text and tables, the objective is to… ibm-research/data-product-benchmark 是 ModelScope 魔搭社区上的数据集，存储大小 876 MB，采用 mit 许可。 - **Repository**: ibm-research/data-product-benchmark - **License**: mit - **Storage size**: 876 MB - **Downloads**: 294 - **Stars**: 0 - **Last updated**: 2026-03-19 Source: https://www.modelscope.cn/datasets/ibm-research/data-product-benchmark --- # DPDisc Dataset [Paper](https://huggingface.co/papers/2510.21737) | [Code](https://github.com/ibm/data-product-benchmark) ### Dataset Description This dataset provides a benchmark for automatic data product creation. The task is framed as follows: given a natural language data product request and a corpus of text and tables, the objective is to identify the relevant tables and text documents that should be included in the resulting data product which would useful to the given data product request. The benchmark brings together three variants: HybridQA, TAT-QA, and ConvFinQA, each consisting of: - A corpus of text passages and tables, and - A set of data product requests along with their corresponding ground-truth tables and text. This benchmark enables systematic evaluation of approaches for discovering tables and text for automatic creation of data products from data lakes with tables and text. ## Dataset Details - **Repository:** https://github.com/ibm/data-product-benchmark - **Paper:** [From Factoid Questions to Data Product Requests: Benchmarking Data Product Discovery over Tables and Text](https://huggingface.co/papers/2510.21737) ### Curation Rationale Data products are reusable, self-contained assets designed for specific business use cases. Automating their discovery and generation is of great industry interest, as it enables discovery in large data lakes and supports analytical Data Product Requests (DPRs). Currently, there is no benchmark established specifically for data product discovery. Existing datasets focus on answering single factoid questions over individual tables rather than collecting multiple data assets for broader, coherent products. To address this gap, we introduce DPDisc, the first user-request-driven data product benchmark over hybrid table-text corpora. Our framework systematically repurposes existing table-text QA datasets such as ConvFinQA, TATQA and HybridQA by clustering related tables and passages into coherent data products, generating professional-level analytical requests that span both data sources, and validating benchmark quality through multi-LLM evaluation. ### Source Datasets | Dataset | Paper | Links | |-----------|-------|-------| | **HybridQA** | [*HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data* (Chen et al., EMNLP Findings 2020)](https://aclanthology.org/2020.findings-emnlp.91/) | [GitHub](https://github.com/wenhuchen/HybridQA) -- [Website](https://hybridqa.github.io/)| | **TAT-QA** | [*TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance* (Zhu et al., ACL-IJCNLP 2021)](https://aclanthology.org/2021.acl-long.254/) | [GitHub](https://github.com/NExTplusplus/TAT-QA) -- [Website](https://nextplusplus.github.io/TAT-QA/)| | **ConvFinQA** | [*ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering* (Chen et al., EMNLP 2022)](https://aclanthology.org/2022.emnlp-main.421/) | [GitHub](https://github.com/czyssrs/ConvFinQA) | ## Dataset Structure ``` benchmark_data/ ├── ConvFinQA/ │ ├── ConvFinQA_dev.jsonl # DPRs + ground truth DPs │ ├── ConvFinQA_test.jsonl │ └── ConvFinQA_train.jsonl │ └── ConvFinQA-corpus/ | └── ConvFinQA_corpus.jsonl # text + table corpora ├── HybridQA/ │ ├── HybridQA_dev.jsonl │ ├── HybridQA_test.jsonl │ └── HybridQA_train.jsonl │ └── HybridQA-corpus/ | └── HybirdQA_corpus.jsonl └── TATQA/ ├── TATQA_dev.jsonl ├── TATQA_test.jsonl └── TATQA_train.jsonl └── TATQA-corpus/ | └── TATQA_corpus.jsonl ``` #### Data Collection and Processing We propose repurposing traditional table–text QA datasets to construct new benchmarks for data product discovery. Rather than focusing on single-table QA, we reinterpret these resources at the table level. By clustering similar QA pairs across multiple tables and their associated passages, we simulate broader data products. We then generate high-level Data Product Requests (DPRs) that abstract away from the low-level questions, while the associated tables and passages serve as ground-truth data products. This reframing enables us to systematically transform QA datasets into DPR benchmarks, providing a cost-effective, scalable alternative to manual construction. ### Benchmark statistics | Dataset | Split | # of DPRs | # of Tables | # of Text Passages | |-----------|-------|-------------|---------------|----------------------| | | Train | 4843 | 12378 | 41,608 | | **Hybrid QA** | Dev | 2008 | ↑| ↑| | | Test | 1980 | ↑| ↑| |||||| | | Train | 820 | 2757 | 4,760 | | **TAT-QA** | Dev | 147 | ↑| ↑| | | Test | 176 | ↑| ↑| |||||| | | Train | 2113 | 4976 | 8721 | | **ConvFinQA** | Dev | 373 | ↑| ↑| | | Test | 627 | ↑| ↑| ## Sample Usage To run the baseline experiments, first clone the repository, set up the Python environment, and then execute the provided script: ```bash # Clone the repository git clone https://github.com/ibm/data-product-benchmark.git cd data-product-benchmark/baselines # Create a Python virtual environment and install dependencies python -m venv venv source venv/bin/activate pip install -r requirements.txt # Run baseline experiments # Ensure the run_baseline.sh script is executable: chmod +x scripts/run_baseline.sh ./scripts/run_baseline.sh ``` Running the baseline script will proceed with producing embeddings and running baseline retrieval methods for a single database at a time. The choice of which data will be used to produce results, as well as the choice of embedding model, can be changed within the `run_baselines.sh` script. Results will be output to files such as `baselines/data/ConfFinQA/ConfFinQA_test_results_eval_granite.json`. > **Note:** HybridQA is the largest of the datasets, and running this baseline may be slow (running locally with no GPU, producing the entire collection of text embeddings may take about an hour). Embedding speed will be much faster if you are running on a machine with GPU support. Producing baseline results for TATQA and ConvFinQA is expected to finish within a few minutes. For more detailed instructions and configuration options, please refer to the [GitHub repository](https://github.com/ibm/data-product-benchmark) under the "Running the baselines" section. ## Citation If you use this work in your research, please cite the [following paper](https://arxiv.org/abs/2510.21737): **BibTeX:** ```bibtex @misc{zhang2025dpr, title={From Factoid Questions to Data Product Requests: Benchmarking Data Product Discovery over Tables and Text}, author={Liangliang Zhang and Nandana Mihindukulasooriya and Niharika S. D'Souza and Sola Shirai and Sarthak Dash and Yao Ma and Horst Samulowitz}, year={2025}, url={https://arxiv.org/abs/2510.21737}, } ```