---
title: scalingcapabilitytokenspace
canonical_url: "https://www.modelscope.cn/models/LiTenghui/scalingcapabilitytokenspace"
md_url: "https://www.modelscope.cn/models/LiTenghui/scalingcapabilitytokenspace.md"
repository: LiTenghui/scalingcapabilitytokenspace
last_updated: 2026-03-27
license: "MIT License"
parameters: 185.7B
tensor_type:
  - BF16
library_name:
  - safetensors
language:
  - en
downloads: 148
stars: 0
---

# scalingcapabilitytokenspace

> scalingcapabilitytokenspace - LiTenghui 在 ModelScope 开源的模型。This is an additional weights for paper: Scaling Capability in Token Space: An Analysis of Large Vision Language Model

LiTenghui/scalingcapabilitytokenspace 是 ModelScope 魔搭社区上的 185.7B 参数机器学习模型，采用 MIT License 许可。

- **Repository**: LiTenghui/scalingcapabilitytokenspace
- **License**: MIT License
- **Parameters**: 185.7B
- **Downloads**: 148
- **Stars**: 0
- **Last updated**: 2026-03-27

Source: https://www.modelscope.cn/models/LiTenghui/scalingcapabilitytokenspace

---

# Scaling Capability in Token Space: Tensor Fusion VLM

This model is part of the research on "Scaling Capability in Token Space: An Analysis of Large Vision Language Model". This research investigates the scaling relationship in vision-language models with respect to the number of vision tokens, revealing theoretical foundations for token-efficient model design.

## Model Description
- Model Type: Vision-Language Model
- Architecture: Tensor Fusion VLM with multi-query image modeling
- Base LLM: Llama-2
- Vision Encoder: CLIP-based encoder
- Vision Tokens: Variable (1, 8, 16, 32, 64, 128, 256, 384, 512, 768 depending on variant)

## Research Overview
Large language models have demonstrated predictable scaling behaviors with respect to model parameters and training data. This study investigates whether a similar scaling relationship exists for vision-language models with respect to the number of vision tokens. Our theoretical analysis reveals two distinct scaling regimes:
- Sublinear scaling for fewer vision tokens
- Linear scaling for more vision tokens

This aligns with model performance relationships of the form $S(n_l) \approx c / n_l^{\alpha}$, where the scaling exponent relates to the correlation structure between vision token representations.

## Available Model Variants
This repository contains multiple model variants with different numbers of vision tokens, organized in separate subdirectories:

- `tensorfusionvlm-v_img_multi_que_clip_fusion_linear-{N}_8_hd_llama-2_eye-ft-finetune`: Models with N vision tokens ($N \in \{1, 8, 16, 32, 64, 128, 256, 384, 512, 768\}$) using Llama-2 as base LLM
- `tensorfusionvlm-v_img_multi_que_clip_fusion_linear-{N}_8_hd_llama-2_drop-query-ft-finetune`: Models with drop-query fine-tuning

Each subdirectory contains a complete model checkpoint with configuration files, tokenizer files, and model weights in safetensors format.


## Training Details
Models were trained using various vision token counts to investigate scaling properties. Each variant was fine-tuned using specific methodologies to optimize performance for its token count.

## File Structure
Each model variant contains:
- `config.json`: Model configuration
- `model-*.safetensors`: Sharded model weights in safetensors format
- `model.safetensors.index.json`: Index file mapping weights to shards
- `tokenizer.json` and `tokenizer.model`: Tokenizer files
- `special_tokens_map.json`, `tokenizer_config.json`: Tokenizer configuration
- `preprocessor_config.json`: Image preprocessing configuration

## Citation
If you use this model in your research, please cite:

```
@article{Li2025ScaCapIn,
  title = {Scaling Capability in Token Space: An Analysis of Large Vision Language Model},
  author = {Tenghui Li and Guoxu Zhou and Xuyang Zhao and Qibin Zhao},
  journal = {Journal of Machine Learning Research},
  year = {2025},
  month = {10},
  url = {http://jmlr.org/papers/v26/24-2443.html},
}
```

## License
MIT License