--- title: Speech-MASSIVE-test canonical_url: "https://www.modelscope.cn/datasets/FBK-MT/Speech-MASSIVE-test" md_url: "https://www.modelscope.cn/datasets/FBK-MT/Speech-MASSIVE-test.md" repository: FBK-MT/Speech-MASSIVE-test last_updated: 2025-09-26 license: "[cc-by-nc-sa-4.0]" storage_size: "22 GB" downloads: 529 stars: 0 --- # Speech-MASSIVE-test > Speech-MASSIVE-test - FBK-MT 在 ModelScope 开源的数据集。Speech-MASSIVE Test Split FBK-MT/Speech-MASSIVE-test 是 ModelScope 魔搭社区上的数据集，存储大小 22 GB，采用 [cc-by-nc-sa-4.0] 许可。 - **Repository**: FBK-MT/Speech-MASSIVE-test - **License**: [cc-by-nc-sa-4.0] - **Storage size**: 22 GB - **Downloads**: 529 - **Stars**: 0 - **Last updated**: 2025-09-26 Source: https://www.modelscope.cn/datasets/FBK-MT/Speech-MASSIVE-test --- # Speech-MASSIVE Test Split This dataset repository is **_only_** for `test` split of Speech-MASSIVE. `train` and `dev` splits are available in the separate dataset repository. [https://huggingface.co/datasets/FBK-MT/Speech-MASSIVE](https://huggingface.co/datasets/FBK-MT/Speech-MASSIVE) ## Dataset Description Speech-MASSIVE is a multilingual Spoken Language Understanding (SLU) dataset comprising the speech counterpart for a portion of the [MASSIVE](https://aclanthology.org/2023.acl-long.235) textual corpus. Speech-MASSIVE covers 12 languages (Arabic, German, Spanish, French, Hungarian, Korean, Dutch, Polish, European Portuguese, Russian, Turkish, and Vietnamese) from different families and inherits from MASSIVE the annotations for the intent prediction and slot-filling tasks. MASSIVE utterances' labels span 18 domains, with 60 intents and 55 slots. Full train split is provided for French and German, and for all the 12 languages (including French and German), we provide few-shot train, dev, test splits. Few-shot train (115 examples) covers all 18 domains, 60 intents, and 55 slots (including empty slots). Our extension is prompted by the scarcity of massively multilingual SLU datasets and the growing need for versatile speech datasets to assess foundation models (LLMs, speech encoders) across diverse languages and tasks. To facilitate speech technology advancements, we release Speech-MASSIVE publicly available with [CC-BY-NC-SA-4.0 license](https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en). Speech-MASSIVE is accepted at INTERSPEECH 2024 (Kos, GREECE) and nominated for the ISCA best student paper award. ## Dataset Summary - `test`: test split available for all the 12 languages - ⚠️ `dev`, `train_115`, `train` splits are available in the separate dataset repository. [https://huggingface.co/datasets/FBK-MT/Speech-MASSIVE](https://huggingface.co/datasets/FBK-MT/Speech-MASSIVE) ⚠️ - `dev`: dev split available for all the 12 languages - `train_115`: few-shot split available for all the 12 languages (all 115 samples are cross-lingually aligned) - `train`: train split available for French (fr-FR) and German (de-DE) | lang | split | # sample | # hrs | total # spk
(Male/Female/Unidentified) | |:---:|:---:|:---:|:---:|:---:| | ar-SA | dev | 2033 | 2.12 | 36 (22/14/0) | | | test | 2974 | 3.23 | 37 (15/17/5) | | | train_115 | 115 | 0.14 | 8 (4/4/0) | | de-DE | dev | 2033 | 2.33 | 68 (35/32/1) | | | test | 2974 | 3.41 | 82 (36/36/10) | | | train | 11514 | 12.61 | 117 (50/63/4) | | | train_115 | 115 | 0.15 | 7 (3/4/0) | | es-ES | dev | 2033 | 2.53 | 109 (51/53/5) | | | test | 2974 | 3.61 | 85 (37/33/15) | | | train_115 | 115 | 0.13 | 7 (3/4/0) | | fr-FR | dev | 2033 | 2.20 | 55 (26/26/3) | | | test | 2974 | 2.65 | 75 (31/35/9) | | | train | 11514 | 12.42 | 103 (50/52/1) | | | train_115 | 115 | 0.12 | 103 (50/52/1) | | hu-HU | dev | 2033 | 2.27 | 69 (33/33/3) | | | test | 2974 | 3.30 | 55 (25/24/6) | | | train_115 | 115 | 0.12 | 8 (3/4/1) | | ko-KR | dev | 2033 | 2.12 | 21 (8/13/0) | | | test | 2974 | 2.66 | 31 (10/18/3) | | | train_115 | 115 | 0.14 | 8 (4/4/0) | | nl-NL | dev | 2033 | 2.14 | 37 (17/19/1) | | | test | 2974 | 3.30 | 100 (48/49/3) | | | train_115 | 115 | 0.12 | 7 (3/4/0) | | pl-PL | dev | 2033 | 2.24 | 105 (50/52/3) | | | test | 2974 | 3.21 | 151 (73/71/7) | | | train_115 | 115 | 0.10 | 7 (3/4/0) | | pt-PT | dev | 2033 | 2.20 | 107 (51/53/3) | | | test | 2974 | 3.25 | 102 (48/50/4) | | | train_115 | 115 | 0.12 | 8 (4/4/0) | | ru-RU | dev | 2033 | 2.25 | 40 (7/31/2) | | | test | 2974 | 3.44 | 51 (25/23/3) | | | train_115 | 115 | 0.12 | 7 (3/4/0) | | tr-TR | dev | 2033 | 2.17 | 71 (36/34/1) | | | test | 2974 | 3.00 | 42 (17/18/7) | | | train_115 | 115 | 0.11 | 6 (3/3/0) | | vi-VN | dev | 2033 | 2.10 | 28 (13/14/1) | | | test | 2974 | 3.23 | 30 (11/14/5) | || train_115 | 115 | 0.11 | 7 (2/4/1) | ## How to use ### How to use The `datasets` library allows you to load and pre-process your dataset in pure Python, at scale. The dataset can be downloaded and prepared in one call to your local drive by using the `load_dataset` function. For example, to download the French config, simply specify the corresponding language config name (i.e., "fr-FR" for French): ```python from datasets import load_dataset speech_massive_fr_test = load_dataset("FBK-MT/Speech-MASSIVE-test", "fr-FR", split="test", trust_remote_code=True) ``` In case you don't have enough space in the machine, you can stream dataset by adding a `streaming=True` argument to the `load_dataset` function call. Loading a dataset in streaming mode loads individual samples of the dataset at a time, rather than downloading the entire dataset to disk. ```python from datasets import load_dataset speech_massive_de_test = load_dataset("FBK-MT/Speech-MASSIVE-test", "de-DE", split="test", streaming=True, trust_remote_code=True) list(speech_massive_de_test.take(2)) ``` You can also load all the available languages and splits at once. And then access each split. ```python from datasets import load_dataset speech_massive = load_dataset("FBK-MT/Speech-MASSIVE-test", "all", trust_remote_code=True) multilingual_test = speech_massive['test'] ``` ## Dataset Structure ### Data configs - `all`: load all the 12 languages in one single dataset instance - `lang`: load only `lang` in the dataset instance, by specifying one of below languages - ```ar-SA, de-DE, es-ES, fr-FR, hu-HU, ko-KR, nl-NL, pl-PL, pt-PT, ru-RU, tr-TR, vi-VN``` ### Data Splits - `test`: test split available for all the 12 languages > [!WARNING] > `validation`, `train_115` and `train` splits are uploaded to a separate dataset repository. - `validation`: validation(dev) split available for all the 12 languages - `train_115`: few-shot (115 samples) split available for all the 12 languages - `train`: train split available for French (fr-FR) and German (de-DE) ### Data Instances ```json { // Start of the data collected in Speech-MASSIVE 'audio': { 'path': 'train/2b12a21ca64a729ccdabbde76a8f8d56.wav', 'array': array([-7.80913979e-...7259e-03]), 'sampling_rate': 16000}, 'path': '/path/to/wav/file.wav', 'is_transcript_reported': False, 'is_validated': True, 'speaker_id': '60fcc09cb546eee814672f44', 'speaker_sex': 'Female', 'speaker_age': '25', 'speaker_ethnicity_simple': 'White', 'speaker_country_of_birth': 'France', 'speaker_country_of_residence': 'Ireland', 'speaker_nationality': 'France', 'speaker_first_language': 'French', // End of the data collected in Speech-MASSIVE // Start of the data extracted from MASSIVE // (https://huggingface.co/datasets/AmazonScience/massive/blob/main/README.md#data-instances) 'id': '7509', 'locale': 'fr-FR', 'partition': 'train', 'scenario': 2, 'scenario_str': 'calendar', 'intent_idx': 32, 'intent_str': 'calendar_query', 'utt': 'après les cours de natation quoi d autre sur mon calendrier mardi', 'annot_utt': 'après les cours de natation quoi d autre sur mon calendrier [date : mardi]', 'worker_id': '22', 'slot_method': {'slot': ['date'], 'method': ['translation']}, 'judgments': { 'worker_id': ['22', '19', '0'], 'intent_score': [1, 2, 1], 'slots_score': [1, 1, 1], 'grammar_score': [4, 4, 4], 'spelling_score': [2, 1, 2], 'language_identification': ['target', 'target', 'target'] }, 'tokens': ['après', 'les', 'cours', 'de', 'natation', 'quoi', 'd', 'autre', 'sur', 'mon', 'calendrier', 'mardi'], 'labels': ['Other', 'Other', 'Other', 'Other', 'Other', 'Other', 'Other', 'Other', 'Other', 'Other', 'Other', 'date'], // End of the data extracted from MASSIVE } ``` ### Data Fields `audio.path`: Original audio file name `audio.array`: Read audio file with the sampling rate of 16,000 `audio.sampling_rate`: Sampling rate `path`: Original audio file full path `is_transcript_reported`: Whether the transcript is reported as 'syntatically wrong' by crowd-source worker `is_validated`: Whether the recorded audio has been validated to check if the audio matches transcript exactly by crowd-source worker `speaker_id`: Unique hash id of the crowd source speaker `speaker_sex`: Speaker's sex information provided by the crowd-source platform ([Prolific](http://prolific.com)) - Male - Female - Unidentified : Information not available from Prolific `speaker_age`: Speaker's age information provided by Prolific - age value (`str`) - Unidentified : Information not available from Prolific `speaker_ethnicity_simple`: Speaker's ethnicity information provided by Prolific - ethnicity value (`str`) - Unidentified : Information not available from Prolific `speaker_country_of_birth`: Speaker's country of birth information provided by Prolific - country value (`str`) - Unidentified : Information not available from Prolific `speaker_country_of_residence`: Speaker's country of residence information provided by Prolific - country value (`str`) - Unidentified : Information not available from Prolific `speaker_nationality`: Speaker's nationality information provided by Prolific - nationality value (`str`) - Unidentified : Information not available from Prolific `speaker_first_language`: Speaker's first language information provided by Prolific - language value (`str`) - Unidentified : Information not available from Prolific ### Limitations As Speech-MASSIVE is constructed based on the MASSIVE dataset, it inherently retains certain grammatical errors present in the original MASSIVE text. Correcting these errors was outside the scope of our project. However, by providing the `is_transcripted_reported` attribute in Speech-MASSIVE, we enable users of the dataset to be aware of these errors. ## License All datasets are licensed under the [CC-BY-NC-SA-4.0 license](https://creativecommons.org/licenses/by-nc-sa/4.0/deed.en). ### Citation Information Speech-MASSIVE is accepted at INTERSPEECH 2024 (Kos, Greece). You can access the Speech-MASSIVE paper on [Interspeech archive](https://www.isca-archive.org/interspeech_2024/lee24i_interspeech.html) or [arXiv (same paper content but with appendix)](https://arxiv.org/abs/2408.03900). Please cite the paper when referencing the Speech-MASSIVE corpus as: ``` @inproceedings{lee24i_interspeech, title = {{Speech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyond}}, author = {Beomseok Lee and Ioan Calapodescu and Marco Gaido and Matteo Negri and Laurent Besacier}, year = {2024}, booktitle = {{Interspeech 2024}}, pages = {817--821}, doi = {10.21437/Interspeech.2024-957}, issn = {2958-1796}, } ```