---
title: pixelprose
canonical_url: "https://www.modelscope.cn/datasets/swift/pixelprose"
md_url: "https://www.modelscope.cn/datasets/swift/pixelprose.md"
repository: swift/pixelprose
last_updated: 2025-12-13
license: cc-by-4.0
storage_size: "9.1 GB"
downloads: 3404
stars: 0
---

# pixelprose

> pixelprose - swift 在 ModelScope 开源的数据集。From Pixels to Prose: A Large Dataset of Dense Image Captions

swift/pixelprose 是 ModelScope 魔搭社区上的数据集，存储大小 9.1 GB，采用 cc-by-4.0 许可。

- **Repository**: swift/pixelprose
- **License**: cc-by-4.0
- **Storage size**: 9.1 GB
- **Downloads**: 3404
- **Stars**: 0
- **Last updated**: 2025-12-13

Source: https://www.modelscope.cn/datasets/swift/pixelprose

---

# From Pixels to Prose: A Large Dataset of Dense Image Captions

[[ **arXiv paper** ](https://arxiv.org/abs/2406.10328)] ｜ [[ 🌮 **image tars** ](https://huggingface.co/datasets/pixelprose/pixelprose-shards)]

**PixelProse** is a comprehensive dataset of over **16M (million)** synthetically generated captions, 
leveraging cutting-edge vision-language models ([Gemini 1.0 Pro Vision](https://console.cloud.google.com/vertex-ai/publishers/google/model-garden/gemini-pro-vision)) for detailed and accurate descriptions.

## 1. Details

Total number of image-caption pairs: 16,896,214 (16.9M)
- 6,538,898 (6.5M) pairs in the split of [CommonPool](https://www.datacomp.ai)
- 9,066,455 (9.1M) pairs in the split of [CC12M](https://github.com/google-research-datasets/conceptual-12m)
- 1,290,861 (1.3M) pairs in the split of [RedCaps](https://redcaps.xyz)

## 2. Download Parquet Files

The first step is to download the parquet files, containing image URLs, captions, and other variables (please check out Dataset Viewer in this repo.)
Three ways to download the parquet files are:

#### via Git LFS

```bash
# make sure you have git-lfs installed (https://git-lfs.com)
git lfs install

# w/ HTTPS
git clone https://huggingface.co/datasets/tomg-group-umd/pixelprose

# w/ SSH
git clone git@hf.co:datasets/tomg-group-umd/pixelprose
```

#### via Huggingface API

```python
from datasets import load_dataset

# for downloading the whole data
ds = load_dataset("tomg-group-umd/pixelprose")

# for downloading specific split
ds_commom_pool = load_dataset("tomg-group-umd/pixelprose", split="commonpool")
ds_cc12m       = load_dataset("tomg-group-umd/pixelprose", split="cc12m")
ds_redcaps     = load_dataset("tomg-group-umd/pixelprose", split="redcaps")
```

The Parquet files are stored in the Hugging Face cache directory, which is located by default at `~/.cache/huggingface/datasets`.
More info can be found [cache management](https://huggingface.co/docs/datasets/en/cache).

#### via Direct Link

Please navigate to the [data](https://huggingface.co/datasets/tomg-group-umd/pixelprose/tree/main/data) directory and click the required parquet file to download.

## 3. Download Images

The second step is to download images using the parquet files. An optional tool for this is [img2dataset](https://github.com/rom1504/img2dataset/tree/main).

You might wanna go to  [[ 🌮 **image tars** ](https://huggingface.co/datasets/pixelprose/pixelprose-shards)] for grabing images.

## 4. Variables

PixelProse has multiple variable columns, which are

- `uid`: unique identifier for the image
- `url`: URL of the image
- `key`: key associated with the image
- `status`: status returned from the `vlm_model`
- `original_caption`: caption inherited from the source
- `vlm_model`: model used for captioning the image
- `vlm_caption`: PixelProse's dense caption
- `toxicity`: score for general toxic behavior or language
- `severe_toxicity`: score for extremely harmful and abusive language
- `obscene`: score for use of obscene or inappropriate language
- `identity_attack`: score for language targeting individuals or groups based on identity
- `insult`: score for language intended to insult or demean
- `threat`: score for language conveying threats of harm
- `sexual_explicit`: score for language with sexually explicit content
- `watermark_class_id`: watermark classification (`0` = image with watermark, `1` = image without watermark, `2` = image without watermark but with text).
- `watermark_class_score`: prediction score for each watermark class, ranging from `[0, 1]`
- `aesthetic_score`: aesthetic score ranging from `[0, 10]`
- `error_message`: error message returned from the `vlm_model`
- `width / height`: size of the image downloaded and used for running the `vlm_model`
- `original_width / original_height`: original size of the image
- `exif`: EXIF information of the image file
- `sha256`: SHA256 hash of the image file
- `image_id`, `author`, `subreddit`, `score`: attributes inherited from RedCaps, unavailable in CC12M and CommonPool

## 5. Contact

If you have any questions about PixelProse, please open a discussion. 
Contributions via pull requests are also welcome.
