---
title: IMAGE_UNDERSTANDING
canonical_url: "https://www.modelscope.cn/datasets/microsoft/IMAGE_UNDERSTANDING"
md_url: "https://www.modelscope.cn/datasets/microsoft/IMAGE_UNDERSTANDING.md"
repository: microsoft/IMAGE_UNDERSTANDING
last_updated: 2025-07-21
license: cdla-permissive-2.0
storage_size: "398 MB"
downloads: 288
stars: 0
---

# IMAGE_UNDERSTANDING

> IMAGE_UNDERSTANDING - microsoft 在 ModelScope 开源的数据集。A key question for understanding multimodal performance is analyzing the ability for a model to have basic vs. detailed understanding of images. These capabilities are needed for models to be used in…

microsoft/IMAGE_UNDERSTANDING 是 ModelScope 魔搭社区上的数据集，存储大小 398 MB，采用 cdla-permissive-2.0 许可。

- **Repository**: microsoft/IMAGE_UNDERSTANDING
- **License**: cdla-permissive-2.0
- **Storage size**: 398 MB
- **Downloads**: 288
- **Stars**: 0
- **Last updated**: 2025-07-21

Source: https://www.modelscope.cn/datasets/microsoft/IMAGE_UNDERSTANDING

---

A key question for understanding multimodal performance is analyzing the ability for a model to have basic 
vs. detailed understanding of images. These capabilities are needed for models to be used in
real-world tasks, such as an assistant in the physical world. While there are many dataset for object detection
and recognition, there are few that test spatial reasoning and other more targeted task such as visual prompting.
The datasets that do exist are static and publicly available, thus there is concern that current AI models could
be trained on these datasets, which makes evaluation with them unreliable. Thus we created a dataset that is
procedurally generated and synthetic, and tests spatial reasoning, visual prompting, as well as object recognition
and detection. The datasets are challenging for most AI models and by being procedurally generated the
benchmark can be regenerated ad infinitum to create new test sets to combat the effects of models being trained
on this data and the results being due to memorization.

This dataset has 4 sub-tasks: Object Recognition, Visual Prompting. Spatial Rea-
soning, and Object Detection. 

For each sub-task, the images consist of images of pasted objects on random
images. The objects are from the COCO object list and are gathered from internet data. Each object is
masked using the DeepLabV3 object detection model and then pasted on a random background from the
Places365 dataset.  The objects are pasted in one of four locations, top, left, bottom, and right, with small
amounts of random rotation, positional jitter, and scale.

There are 2 conditions “ single” and “ pairs”, for images with one and two objects. Each test set uses 20
sets of object classes (either 20 single objects or 20 pairs of objects), with four potential locations and four
backgrounds classes, and we sample 4 instances of object and background. This results in 1280 images per
condition and sub-task.

__Object Detection__

	Answer type: Open-ended

	Example for "single":

		{"images": ["val\\banana\\left\\fire_station\\0000075_Places365_val_00030609.jpg"], "prompt": "You are an object detection model that aims to detect all the objects in the image.\n\nDefinition of Bounding Box Coordinates:\n\nThe bounding box coordinates (a, b, c, d) represent the normalized positions of the object within the image:\n\na: The x-coordinate of the top-left corner of the bounding box, expressed as a percentage of the image width. It indicates the position from the left side of the image to the object's left boundary. The a ranges from 0.00 to 1.00 with precision of 0.01.\nb: The y-coordinate of the top-left corner of the bounding box, expressed as a percentage of the image height. It indicates the position from the top of the image to the object's top boundary. The b ranges from 0.00 to 1.00 with precision of 0.01.\nc: The x-coordinate of the bottom-right corner of the bounding box, expressed as a percentage of the image width. It indicates the position from the left side of the image to the object's right boundary. The c ranges from 0.00 to 1.00 with precision of 0.01.\nd: The y-coordinate of the bottom-right corner of the bounding box, expressed as a percentage of the image height. It indicates the position from the top of the image to the object's bottom boundary. The d ranges from 0.00 to 1.00 with precision of 0.01.\n\nThe top-left of the image has coordinates (0.00, 0.00). The bottom-right of the image has coordinates (1.00, 1.00).\n\nInstructions:\n1. Specify any particular regions of interest within the image that should be prioritized during object detection.\n2. For all the specified regions that contain the objects, generate the object's category type, bounding box coordinates, and your confidence for the prediction. The bounding box coordinates (a, b, c, d) should be as precise as possible. Do not only output rough coordinates such as (0.1, 0.2, 0.3, 0.4).\n3. If there are more than one object of the same category, output all of them.\n4. Please ensure that the bounding box coordinates are not examples. They should really reflect the position of the objects in the image.\n5.\nReport your results in this output format:\n(a, b, c, d) - category for object 1 - confidence\n(a, b, c, d) - category for object 2 - confidence\n...\n(a, b, c, d) - category for object n - confidence."}

	Example for "pairs":

		{"images": ["val\\hair drier_broccoli\\left\\church-indoor\\0000030_0000059_Places365_val_00000401.jpg"], "prompt": "You are an object detection model that aims to detect all the objects in the image.\n\nDefinition of Bounding Box Coordinates:\n\nThe bounding box coordinates (a, b, c, d) represent the normalized positions of the object within the image:\n\na: The x-coordinate of the top-left corner of the bounding box, expressed as a percentage of the image width. It indicates the position from the left side of the image to the object's left boundary. The a ranges from 0.00 to 1.00 with precision of 0.01.\nb: The y-coordinate of the top-left corner of the bounding box, expressed as a percentage of the image height. It indicates the position from the top of the image to the object's top boundary. The b ranges from 0.00 to 1.00 with precision of 0.01.\nc: The x-coordinate of the bottom-right corner of the bounding box, expressed as a percentage of the image width. It indicates the position from the left side of the image to the object's right boundary. The c ranges from 0.00 to 1.00 with precision of 0.01.\nd: The y-coordinate of the bottom-right corner of the bounding box, expressed as a percentage of the image height. It indicates the position from the top of the image to the object's bottom boundary. The d ranges from 0.00 to 1.00 with precision of 0.01.\n\nThe top-left of the image has coordinates (0.00, 0.00). The bottom-right of the image has coordinates (1.00, 1.00).\n\nInstructions:\n1. Specify any particular regions of interest within the image that should be prioritized during object detection.\n2. For all the specified regions that contain the objects, generate the object's category type, bounding box coordinates, and your confidence for the prediction. The bounding box coordinates (a, b, c, d) should be as precise as possible. Do not only output rough coordinates such as (0.1, 0.2, 0.3, 0.4).\n3. If there are more than one object of the same category, output all of them.\n4. Please ensure that the bounding box coordinates are not examples. They should really reflect the position of the objects in the image.\n5.\nReport your results in this output format:\n(a, b, c, d) - category for object 1 - confidence\n(a, b, c, d) - category for object 2 - confidence\n...\n(a, b, c, d) - category for object n - confidence."}

__Object Recognition__

	Answer type: Open-ended

	Example for "single"

		{"images": ["val\\potted plant\\left\\ruin\\0000097_Places365_val_00018147.jpg"], "prompt": "What objects are in this image?", "ground_truth": "potted plant"}

	Example for "pairs":

		{"images": ["val\\bottle_keyboard\\left\\ruin\\0000087_0000069_Places365_val_00035062.jpg"], "prompt": "What objects are in this image?", "ground_truth": "['bottle', 'keyboard']"}

__Spatial Reasoning__

	Answer type: Multiple Choice

	Example for "single"

		{"images": ["val\\potted plant\\left\\ruin\\0000097_Places365_val_00018147.jpg"], 
		"query_text": "Is the potted plant on the right, top, left, or bottom of the image?\nAnswer with one of (right, bottom, top, or left) only.", 
		"target_text": "left"}

	Example for "pairs"

		{"images": ["val\\bottle_keyboard\\left\\ruin\\0000087_0000069_Places365_val_00035062.jpg"], 
		"query_text": "Is the bottle above, below, right, or left of the keyboard in the image?\nAnswer with one of (below, right, left, or above) only.", 
		"target_text": "left"}

	What are the evaluation disaggregation pivots/attributes to run metrics for?

	Disaggregation by (group by):

	"single": (left, right, top, bottom)
	"pairs": (left, right, above, below) 

__Visual Prompting__

Answer type: Open-ended

	Example for "single"

		{"images": ["val\\potted plant\\left\\ruin\\0000097_Places365_val_00018147.jpg"], "prompt": "What objects are in this image?", "ground_truth": "potted plant"}

	Example for "pairs":

		{"images": ["val\\sheep_banana\\left\\landfill\\0000099_0000001_Places365_val_00031238.jpg"], "prompt": "What objects are in the red and yellow box in this image?", "ground_truth": "['sheep', 'banana']"}