---
title: StructuredCodeRepresentations
canonical_url: "https://www.modelscope.cn/datasets/davidyuan666/StructuredCodeRepresentations"
md_url: "https://www.modelscope.cn/datasets/davidyuan666/StructuredCodeRepresentations.md"
repository: davidyuan666/StructuredCodeRepresentations
chinese_name: "结构化代码表示数据集"
last_updated: 2026-07-10
license: Apache-2.0
storage_size: "9.0 GB"
downloads: 330
stars: 0
---

# StructuredCodeRepresentations

> StructuredCodeRepresentations - davidyuan666 在 ModelScope 开源的数据集。Code Structural Representations Dataset, Preprocessed code structural representations for clone detection and vulnerability detection. Includes three datasets (BCBench, Devign, OJClone) each…

davidyuan666/StructuredCodeRepresentations 是 ModelScope 魔搭社区上的数据集，存储大小 9.0 GB，采用 Apache-2.0 许可。

- **Repository**: davidyuan666/StructuredCodeRepresentations
- **License**: Apache-2.0
- **Storage size**: 9.0 GB
- **Downloads**: 330
- **Stars**: 0
- **Last updated**: 2026-07-10

Source: https://www.modelscope.cn/datasets/davidyuan666/StructuredCodeRepresentations

---

# Structured Code Representations

A preprocessed dataset of structured code representations for code clone detection and vulnerability detection. Contains three benchmark datasets, each with four representation schemes, plus three independent SentencePiece BPE tokenizers (50K vocabulary each).

## Overview

| Dataset | Language | Task | #Samples | #Unique Sources |
|---------|----------|------|----------|-----------------|
| BigCloneBench (BCBench) | Java | Clone Detection | 1,731,860 pairs | 8,063 |
| OJClone (POJ-104) | C | Clone Detection | 106,000 pairs | 51,995 |
| Devign | C | Vulnerability Detection | 27,318 samples | 27,258 |

## Representations

| Name | Description |
|------|-------------|
| `raw` | Whitespace-tokenized source code sequence (baseline) |
| `ast_seq` | AST node sequence via tree-sitter, DFS pre-order, identifiers preserved |
| `cfg_seq` | Control-flow skeleton: ENTRY → BRANCH / LOOP / JUMP / RETURN / FUNC → EXIT |
| `ir_seq` | Statement-level instruction sequence: EXPR, DECL, ASSIGN, CALL, IF, WHILE, ... |

All structural representations (AST, CFG, IR) are extracted via [tree-sitter](https://tree-sitter.github.io/) with bindings for C, C++, and Java. No compilation is required.

## Data Format

Every representation provides `train.jsonl` / `val.jsonl` / `test.jsonl` in a fixed 8:1:1 split.

**Clone detection** (BCBench, OJClone):
```json
{"code_a": "int main ( ) { return 0 ; }", "code_b": "int main ( ) { return 1 ; }", "label": 0}
```

**Vulnerability detection** (Devign):
```json
{"code": "static av_cold int vdadec_init ( AVCodecContext * avctx ) ...", "label": 0}
```

## Tokenizers

Three independent SPM BPE tokenizers (50K vocab each, `identity` normalization):

| File | Trained On | Corpus | Use Case |
|------|-----------|--------|----------|
| `spm_shared_50k.model` | raw + ast + cfg + ir | 54.5M tokens | RQ1a: shared vocabulary |
| `spm_raw_50k.model` | raw code only | 9M tokens | RQ1b: raw-optimized |
| `spm_struct_50k.model` | ast + cfg + ir | 45.5M tokens | RQ1b: structure-optimized |

## Dataset Files

```
processed.zip (2.5 GB)
├── bcbench/raw/                   train.jsonl, val.jsonl, test.jsonl
├── bcbench/ast_seq/               train.jsonl, val.jsonl, test.jsonl
├── bcbench/cfg_seq/               train.jsonl, val.jsonl, test.jsonl
├── bcbench/ir_seq/                train.jsonl, val.jsonl, test.jsonl
├── devign/{raw,ast_seq,cfg_seq,ir_seq}/
├── ojclone/{raw,ast_seq,cfg_seq,ir_seq}/
├── spm_shared_50k.model + .vocab
├── spm_raw_50k.model    + .vocab
└── spm_struct_50k.model + .vocab
```

## Statistics (SPM Shared Tokenizer)

| Representation | OOV Rate | Subwords/Token | Avg Seq Len | Trunc@512 |
|---------------|----------|:---:|:---:|:---:|
| Raw Code | 0.30–1.38% | 2.41–3.34 | 191–508 | 1.2–26.8% |
| AST Sequence | 0.05–0.48% | 1.61–1.89 | 596–1045 | 46.4–51.6% |
| CFG Block Labels | 0.00% | 1.11–1.69 | 15–23 | 0.0–0.3% |
| IR Instruction Seq | 0.00% | 1.00 | 29–90 | 0.1–1.7% |

## Quick Start

```python
from modelscope.msdatasets import MsDataset
ds = MsDataset.load('davidyuan666/StructuredCodeRepresentations')
```

Or download via CLI:

```bash
modelscope download davidyuan666/StructuredCodeRepresentations \
  --files processed.zip --local_dir ./data/
unzip ./data/processed.zip -d ./data/processed/
```

Load tokenizer:

```python
import sentencepiece as spm
sp = spm.SentencePieceProcessor(model_file='data/processed/spm_shared_50k.model')
```

## Paper

```bibtex
@article{yuan2025structured,
  title={Revisiting Structured Code Representations: Serialization, Vocabulary, and Traversal Effects},
  author={Yuan, Dawei and Liang, Guojun and Liu, Bin and Liu, Suping},
  year={2025}
}
```

## Code

[https://github.com/davidyuan666/structured-code-repr-analysis](https://github.com/davidyuan666/structured-code-repr-analysis)

## Sources

| Dataset | HuggingFace |
|---------|-------------|
| BCBench | `google/code_x_glue_cc_clone_detection_big_clone_bench` |
| OJClone | `google/code_x_glue_cc_clone_detection_poj104` |
| Devign | `DetectVul/devign` |

## License

Apache 2.0