--- title: code_x_glue_tc_nl_code_search_adv canonical_url: "https://www.modelscope.cn/datasets/google/code_x_glue_tc_nl_code_search_adv" md_url: "https://www.modelscope.cn/datasets/google/code_x_glue_tc_nl_code_search_adv.md" repository: google/code_x_glue_tc_nl_code_search_adv last_updated: 2025-04-21 license: "Apache License 2.0" storage_size: "302 MB" downloads: 878 stars: 0 --- # code_x_glue_tc_nl_code_search_adv > code_x_glue_tc_nl_code_search_adv - google 在 ModelScope 开源的数据集。Dataset Card for "codexgluetcnlcodesearchadv" google/code_x_glue_tc_nl_code_search_adv 是 ModelScope 魔搭社区上的数据集，存储大小 302 MB，采用 Apache License 2.0 许可。 - **Repository**: google/code_x_glue_tc_nl_code_search_adv - **License**: Apache License 2.0 - **Storage size**: 302 MB - **Downloads**: 878 - **Stars**: 0 - **Last updated**: 2025-04-21 Source: https://www.modelscope.cn/datasets/google/code_x_glue_tc_nl_code_search_adv --- # Dataset Card for "code_x_glue_tc_nl_code_search_adv" ## Table of Contents - [Dataset Description](#dataset-description) - [Dataset Summary](#dataset-summary) - [Supported Tasks and Leaderboards](#supported-tasks) - [Languages](#languages) - [Dataset Structure](#dataset-structure) - [Data Instances](#data-instances) - [Data Fields](#data-fields) - [Data Splits](#data-splits-sample-size) - [Dataset Creation](#dataset-creation) - [Curation Rationale](#curation-rationale) - [Source Data](#source-data) - [Annotations](#annotations) - [Personal and Sensitive Information](#personal-and-sensitive-information) - [Considerations for Using the Data](#considerations-for-using-the-data) - [Social Impact of Dataset](#social-impact-of-dataset) - [Discussion of Biases](#discussion-of-biases) - [Other Known Limitations](#other-known-limitations) - [Additional Information](#additional-information) - [Dataset Curators](#dataset-curators) - [Licensing Information](#licensing-information) - [Citation Information](#citation-information) - [Contributions](#contributions) ## Dataset Description - **Homepage:** https://github.com/microsoft/CodeXGLUE/tree/main/Text-Code/NL-code-search-Adv - **Paper:** https://arxiv.org/abs/2102.04664 ### Dataset Summary CodeXGLUE NL-code-search-Adv dataset, available at https://github.com/microsoft/CodeXGLUE/tree/main/Text-Code/NL-code-search-Adv The dataset we use comes from CodeSearchNet and we filter the dataset as the following: - Remove examples that codes cannot be parsed into an abstract syntax tree. - Remove examples that #tokens of documents is < 3 or >256 - Remove examples that documents contain special tokens (e.g. or https:...) - Remove examples that documents are not English. ### Supported Tasks and Leaderboards - `document-retrieval`: The dataset can be used to train a model for retrieving top-k codes from a given **English** natural language query. ### Languages - Python **programming** language - English **natural** language ## Dataset Structure ### Data Instances An example of 'validation' looks as follows. ``` { "argument_list": "", "code": "def Func(arg_0, arg_1='.', arg_2=True, arg_3=False, **arg_4):\n \"\"\"Downloads Dailymotion videos by URL.\n \"\"\"\n\n arg_5 = get_content(rebuilt_url(arg_0))\n arg_6 = json.loads(match1(arg_5, r'qualities\":({.+?}),\"'))\n arg_7 = match1(arg_5, r'\"video_title\"\\s*:\\s*\"([^\"]+)\"') or \\\n match1(arg_5, r'\"title\"\\s*:\\s*\"([^\"]+)\"')\n arg_7 = unicodize(arg_7)\n\n for arg_8 in ['1080','720','480','380','240','144','auto']:\n try:\n arg_9 = arg_6[arg_8][1][\"url\"]\n if arg_9:\n break\n except KeyError:\n pass\n\n arg_10, arg_11, arg_12 = url_info(arg_9)\n\n print_info(site_info, arg_7, arg_10, arg_12)\n if not arg_3:\n download_urls([arg_9], arg_7, arg_11, arg_12, arg_1=arg_1, arg_2=arg_2)", "code_tokens": ["def", "Func", "(", "arg_0", ",", "arg_1", "=", "'.'", ",", "arg_2", "=", "True", ",", "arg_3", "=", "False", ",", "**", "arg_4", ")", ":", "arg_5", "=", "get_content", "(", "rebuilt_url", "(", "arg_0", ")", ")", "arg_6", "=", "json", ".", "loads", "(", "match1", "(", "arg_5", ",", "r'qualities\":({.+?}),\"'", ")", ")", "arg_7", "=", "match1", "(", "arg_5", ",", "r'\"video_title\"\\s*:\\s*\"([^\"]+)\"'", ")", "or", "match1", "(", "arg_5", ",", "r'\"title\"\\s*:\\s*\"([^\"]+)\"'", ")", "arg_7", "=", "unicodize", "(", "arg_7", ")", "for", "arg_8", "in", "[", "'1080'", ",", "'720'", ",", "'480'", ",", "'380'", ",", "'240'", ",", "'144'", ",", "'auto'", "]", ":", "try", ":", "arg_9", "=", "arg_6", "[", "arg_8", "]", "[", "1", "]", "[", "\"url\"", "]", "if", "arg_9", ":", "break", "except", "KeyError", ":", "pass", "arg_10", ",", "arg_11", ",", "arg_12", "=", "url_info", "(", "arg_9", ")", "print_info", "(", "site_info", ",", "arg_7", ",", "arg_10", ",", "arg_12", ")", "if", "not", "arg_3", ":", "download_urls", "(", "[", "arg_9", "]", ",", "arg_7", ",", "arg_11", ",", "arg_12", ",", "arg_1", "=", "arg_1", ",", "arg_2", "=", "arg_2", ")"], "docstring": "Downloads Dailymotion videos by URL.", "docstring_summary": "Downloads Dailymotion videos by URL.", "docstring_tokens": ["Downloads", "Dailymotion", "videos", "by", "URL", "."], "func_name": "", "id": 0, "identifier": "dailymotion_download", "language": "python", "nwo": "soimort/you-get", "original_string": "", "parameters": "(url, output_dir='.', merge=True, info_only=False, **kwargs)", "path": "src/you_get/extractors/dailymotion.py", "repo": "", "return_statement": "", "score": 0.9997601509094238, "sha": "b746ac01c9f39de94cac2d56f665285b0523b974", "url": "https://github.com/soimort/you-get/blob/b746ac01c9f39de94cac2d56f665285b0523b974/src/you_get/extractors/dailymotion.py#L13-L35" } ``` ### Data Fields In the following each data field in go is explained for each config. The data fields are the same among all splits. #### default | field name | type | description | |-----------------|-----------------------|-----------------------------------------------------------------------------------| |id |int32 | Index of the sample | |repo |string | repo: the owner/repo | |path |string | path: the full path to the original file | |func_name |string | func_name: the function or method name | |original_string |string | original_string: the raw string before tokenization or parsing | |language |string | language: the programming language | |code |string | code/function: the part of the original_string that is code | |code_tokens |Sequence[string] | code_tokens/function_tokens: tokenized version of code | |docstring |string | docstring: the top-level comment or docstring, if it exists in the original string| |docstring_tokens |Sequence[string] | docstring_tokens: tokenized version of docstring | |sha |string | sha of the file | |url |string | url of the file | |docstring_summary|string | Summary of the docstring | |parameters |string | parameters of the function | |return_statement |string | return statement | |argument_list |string | list of arguments of the function | |identifier |string | identifier | |nwo |string | nwo | |score |datasets.Value("float"]| score for this search | ### Data Splits | name |train |validation|test | |-------|-----:|---------:|----:| |default|251820| 9604|19210| ## Dataset Creation ### Curation Rationale [More Information Needed] ### Source Data #### Initial Data Collection and Normalization Data from CodeSearchNet Challenge dataset. [More Information Needed] #### Who are the source language producers? Software Engineering developers. ### Annotations #### Annotation process [More Information Needed] #### Who are the annotators? [More Information Needed] ### Personal and Sensitive Information [More Information Needed] ## Considerations for Using the Data ### Social Impact of Dataset [More Information Needed] ### Discussion of Biases [More Information Needed] ### Other Known Limitations [More Information Needed] ## Additional Information ### Dataset Curators https://github.com/microsoft, https://github.com/madlag ### Licensing Information Computational Use of Data Agreement (C-UDA) License. ### Citation Information ``` @article{DBLP:journals/corr/abs-2102-04664, author = {Shuai Lu and Daya Guo and Shuo Ren and Junjie Huang and Alexey Svyatkovskiy and Ambrosio Blanco and Colin B. Clement and Dawn Drain and Daxin Jiang and Duyu Tang and Ge Li and Lidong Zhou and Linjun Shou and Long Zhou and Michele Tufano and Ming Gong and Ming Zhou and Nan Duan and Neel Sundaresan and Shao Kun Deng and Shengyu Fu and Shujie Liu}, title = {CodeXGLUE: {A} Machine Learning Benchmark Dataset for Code Understanding and Generation}, journal = {CoRR}, volume = {abs/2102.04664}, year = {2021} } @article{husain2019codesearchnet, title={Codesearchnet challenge: Evaluating the state of semantic code search}, author={Husain, Hamel and Wu, Ho-Hsiang and Gazit, Tiferet and Allamanis, Miltiadis and Brockschmidt, Marc}, journal={arXiv preprint arXiv:1909.09436}, year={2019} } ``` ### Contributions Thanks to @madlag (and partly also @ncoop57) for adding this dataset.