--- title: SkyworkVL-38B canonical_url: "https://www.modelscope.cn/models/Skywork/SkyworkVL-38B" md_url: "https://www.modelscope.cn/models/Skywork/SkyworkVL-38B.md" repository: Skywork/SkyworkVL-38B last_updated: 2025-03-18 license: apache-2.0 pipeline_tag: image-text-to-text tasks: - image-text-to-text model_type: - skywork_chat base_model: - OpenGVLab/InternVL2_5-38B base_model_relation: finetune parameters: 38.4B tensor_type: - BF16 library_name: - pytorch - transformer - safetensors frameworks: - pytorch language: - en downloads: 81 stars: 0 --- # SkyworkVL-38B > SkyworkVL-38B - Skywork 在 ModelScope 开源的模型。SkyworkVL-38B: Multimodal Understanding with Bag of Tricks Skywork/SkyworkVL-38B 是 ModelScope 魔搭社区上的 38.4B 参数image-text-to-text模型，采用 apache-2.0 许可，基于 OpenGVLab/InternVL2_5-38B 构建。 - **Repository**: Skywork/SkyworkVL-38B - **License**: apache-2.0 - **Tasks**: image-text-to-text - **Parameters**: 38.4B - **Base model**: OpenGVLab/InternVL2_5-38B - **Downloads**: 81 - **Stars**: 0 - **Last updated**: 2025-03-18 Source: https://www.modelscope.cn/models/Skywork/SkyworkVL-38B --- # SkyworkVL-38B: Multimodal Understanding with Bag of Tricks --- ## Introduction **SkyworkVL-38B** is an advanced VLM model trained on 2 million high-quality caption and QA samples. Leveraging innovative techniques across multiple training stages, our model delivers superior performance on a range of vision-language tasks. ## 🔑 Key Features ### 1. Multi-Resolution Processing - Images are processed at multiple resolutions. For each resolution (from high to low), we apply Closest Aspect Ratio Matching to partition the image into tiles. Finally, the original image is resized into a tile and appended to the final representation—ensuring comprehensive image understanding. ### 2. Multi-Stage Supervised Fine-Tuning (SFT) - **Stage 1:** Fine-tuning on the full dataset. - **Stage 2:** Refinement using a curated subset of 200K high-scoring samples filtered by GPT-4 evaluations. ### 3. High-Quality Chain-of-Thought (CoT) Fine-Tuning - Fine-tuning on 40K high-quality CoT data including self-collected multimodal Chinese Gaokao data with detailed analysis to boost the model’s reasoning capability. ## Model Introduction | Model Name | Base Model | Parameters | Download Link | | ----------------------- | ------------------------- | ---------- | ----------------------------------------------------------- | | SkyworkVL-2B | [OpenGVLab/InternVL2_5-2B](https://huggingface.co/OpenGVLab/InternVL2_5-2B) | 2B | 🤗 [Download](https://huggingface.co/Skywork/SkyworkVL-2B) | | SkyworkVL-38B | [OpenGVLab/InternVL2_5-38B](https://huggingface.co/OpenGVLab/InternVL2_5-38B) | 38B | 🤗 [Download](https://huggingface.co/Skywork/SkyworkVL-38B) | ## Performance | Metric | MathVista (testmini) | MMMU (val) | AI2D | OCRBench | MME | **RealWorldQA** | **HallusionBench** | | --------------------------- | -------------------- | --------------- | --------------- | ------------- | -------------- | --------------- | ------------------ | | Cambrain-34B | 53.2 | 49.7 | 79.5 | 600 | - | 67.8 | 41.6 | | Internvl2-40B | 63.7 | 55.2 | 86.6 | 837 | 2307 | 71.8 | 56.9 | | Internvl2.5-38B | 71.9 | 63.9 | 87.6 | 842 | 2455 | 73.5 | 56.8 | | SkyworkVL-38B | **74.4** | **64.0** | **88.4** | **854** | **2479** | **76.9** | **58.9** | *The performance improvements above demonstrate notable gains in multi-disciplinary question answering, object detection (BBox), and scientific chart analysis among other benchmarks.* ## Usage We provide an example code to run `SkyworkVL-38B` using `transformers` ### Model Loading ```python import torch from transformers import AutoTokenizer, AutoModel path = "Skywork/SkyworkVL-38B" model = AutoModel.from_pretrained( path, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, use_flash_attn=True, trust_remote_code=True).eval().cuda() ``` ### Inference with Transformers ```python import math import numpy as np import torch import torchvision.transforms as T from decord import VideoReader, cpu from PIL import Image from torchvision.transforms.functional import InterpolationMode from transformers import AutoModel, AutoTokenizer IMAGENET_MEAN = (0.485, 0.456, 0.406) IMAGENET_STD = (0.229, 0.224, 0.225) def build_transform(input_size): MEAN, STD = IMAGENET_MEAN, IMAGENET_STD transform = T.Compose([ T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img), T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC), T.ToTensor(), T.Normalize(mean=MEAN, std=STD) ]) return transform def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size): best_ratio_diff = float('inf') best_ratio = (1, 1) area = width * height for ratio in target_ratios: target_aspect_ratio = ratio[0] / ratio[1] ratio_diff = abs(aspect_ratio - target_aspect_ratio) if ratio_diff < best_ratio_diff: best_ratio_diff = ratio_diff best_ratio = ratio elif ratio_diff == best_ratio_diff: if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]: best_ratio = ratio return best_ratio def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False): orig_width, orig_height = image.size aspect_ratio = orig_width / orig_height # calculate the existing image aspect ratio target_ratios = set( (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if i * j <= max_num and i * j >= min_num) target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1]) # find the closest aspect ratio to the target target_aspect_ratio = find_closest_aspect_ratio( aspect_ratio, target_ratios, orig_width, orig_height, image_size) # calculate the target width and height target_width = image_size * target_aspect_ratio[0] target_height = image_size * target_aspect_ratio[1] blocks = target_aspect_ratio[0] * target_aspect_ratio[1] # resize the image resized_img = image.resize((target_width, target_height)) processed_images = [] for i in range(blocks): box = ( (i % (target_width // image_size)) * image_size, (i // (target_width // image_size)) * image_size, ((i % (target_width // image_size)) + 1) * image_size, ((i // (target_width // image_size)) + 1) * image_size ) # split the image split_img = resized_img.crop(box) processed_images.append(split_img) assert len(processed_images) == blocks if use_thumbnail and len(processed_images) != 1: thumbnail_img = image.resize((image_size, image_size)) processed_images.append(thumbnail_img) return processed_images def load_image(image_file, input_size=448, max_num=12): image = Image.open(image_file).convert('RGB') transform = build_transform(input_size=input_size) images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num) pixel_values = [transform(image) for image in images] pixel_values = torch.stack(pixel_values) return pixel_values def split_model(model_name): device_map = {} world_size = torch.cuda.device_count() num_layers = { 'SkyworkVL-2B': 24, 'SkyworkVL-38B': 64}[model_name] num_layers_per_gpu = math.ceil(num_layers / (world_size - 0.5)) num_layers_per_gpu = [num_layers_per_gpu] * world_size num_layers_per_gpu[0] = math.ceil(num_layers_per_gpu[0] * 0.5) layer_cnt = 0 for i, num_layer in enumerate(num_layers_per_gpu): for j in range(num_layer): device_map[f'language_model.model.layers.{layer_cnt}'] = i layer_cnt += 1 device_map['vision_model'] = 0 device_map['mlp1'] = 0 device_map['language_model.model.tok_embeddings'] = 0 device_map['language_model.model.embed_tokens'] = 0 device_map['language_model.output'] = 0 device_map['language_model.model.norm'] = 0 device_map['language_model.model.rotary_emb'] = 0 device_map['language_model.lm_head'] = 0 device_map[f'language_model.model.layers.{num_layers - 1}'] = 0 return device_map path = 'Skywork/SkyworkVL-38B' device_map = split_model('SkyworkVL-38B') model = AutoModel.from_pretrained( path, torch_dtype=torch.bfloat16, load_in_8bit=True, low_cpu_mem_usage=True, use_flash_attn=True, trust_remote_code=True, device_map=device_map).eval() tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False) # set the max number of tiles in `max_num` pixel_values = load_image('./demo/image1.jpg', max_num=12).to(torch.bfloat16).cuda() generation_config = dict(max_new_tokens=1024, do_sample=True) # pure-text conversation (纯文本对话) question = 'Hi, what can you do?' response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True) print(f'User: {question}\nAssistant: {response}') question = 'Can you explain quantum mechanics to me?' response, history = model.chat(tokenizer, None, question, generation_config, history=history, return_history=True) print(f'User: {question}\nAssistant: {response}') # image-text conversation (图文对话) question = '\nWhat do you see in this image?' response = model.chat(tokenizer, pixel_values, question, generation_config) print(f'User: {question}\nAssistant: {response}') ``` ## Citation ```BibTeX @misc{SkyworkVL, author = {Jiangbo Pei and Chris and Yichen Wei and Xiaokun Wang and Yi Peng and Weijie Qiu and Ai Jian and Yunzhuo Hao and Jiachun Pan and Tianyidan Xie and Li Ge and Rongxian Zhuang and Xuchen Song and Yang Liu and Yahui Zhou}, title = {SkyworkVL: Multimodal Understanding with Bag of Tricks}, year = {2025}, publisher = {Huggingface}, journal = {Huggingface repository}, howpublished = {\url{https://huggingface.co/Skywork/SkyworkVL-38B}} } ```