--- title: imp-v1-3b canonical_url: "https://www.modelscope.cn/models/AI-ModelScope/imp-v1-3b" md_url: "https://www.modelscope.cn/models/AI-ModelScope/imp-v1-3b.md" repository: AI-ModelScope/imp-v1-3b last_updated: 2024-02-06 license: apache-2.0 pipeline_tag: text-generation tasks: - text-generation model_type: - imp architectures: - ImpForCausalLM parameters: 3.2B tensor_type: - F16 library_name: - pytorch - safetensors frameworks: - Pytorch downloads: 242 stars: 0 --- # imp-v1-3b > imp-v1-3b - AI-ModelScope 在 ModelScope 开源的模型。A very small man can cast a very large shadow.           ——George R.R. Martin, A Clash of Kings AI-ModelScope/imp-v1-3b 是 ModelScope 魔搭社区上的 3.2B 参数text-generation模型，采用 apache-2.0 许可。 - **Repository**: AI-ModelScope/imp-v1-3b - **License**: apache-2.0 - **Tasks**: text-generation - **Parameters**: 3.2B - **Downloads**: 242 - **Stars**: 0 - **Last updated**: 2024-02-06 Source: https://www.modelscope.cn/models/AI-ModelScope/imp-v1-3b --- # 😈 Imp > A very small man can cast a very large shadow. > >           ——*George R.R. Martin, A Clash of Kings* \[Technical report (coming soon)\]  [[Demo](https://xmbot.net/imp/)\]  [[Github](https://github.com/MILVLG/imp)\] ## Introduction The Imp project aims to provide a family of a strong multimodal `small` language models (MSLMs). Our `imp-v1-3b` is a strong MSLM with only **3B** parameters, which is build upon a small yet powerful SLM [Phi-2 ](https://huggingface.co/microsoft/phi-2)(2.7B) and a powerful visual encoder [SigLIP ](https://huggingface.co/google/siglip-so400m-patch14-384)(0.4B), and trained on the [LLaVA-v1.5](https://github.com/haotian-liu/LLaVA) training set. As shown in the Table below, `imp-v1-3b` significantly outperforms the counterparts of similar model sizes, and even achieves slightly better performance than the strong LLaVA-7B model on various multimodal benchmarks. We release our model weights and provide an example below to run our model . Detailed technical report and corresponding training/evaluation code will be released soon on our [GitHub repo](https://github.com/MILVLG/imp). We will persistently improve our model and release the next versions to further improve model performance :) ## How to use **Install modelscope** ```bash pip install modelscope pip install -q pillow accelerate einops ``` You can use the following code for model inference. The format of text instruction is similar to [LLaVA](https://github.com/haotian-liu/LLaVA). ```Python import torch from modelscope import AutoModelForCausalLM, AutoTokenizer from PIL import Image torch.set_default_device("cuda") #Create model model = AutoModelForCausalLM.from_pretrained( "AI-ModelScope/imp-v1-3b", torch_dtype=torch.float16, device_map="auto", trust_remote_code=True) tokenizer = AutoTokenizer.from_pretrained("AI-ModelScope/imp-v1-3b", trust_remote_code=True) #Set inputs text = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: \nWhat are the colors of the bus in the image? ASSISTANT:" image = Image.open("images/bus.jpg") input_ids = tokenizer(text, return_tensors='pt').input_ids image_tensor = model.image_preprocess(image) #Generate the answer output_ids = model.generate( input_ids, max_new_tokens=100, images=image_tensor, use_cache=True)[0] print(tokenizer.decode(output_ids[input_ids.shape[1]:], skip_special_tokens=True).strip()) ``` ## Model evaluation We conduct evaluation on 9 commonly-used benchmarks, including 5 academic VQA benchmarks and 4 popular MLLM benchmarks, to compare our Imp model with LLaVA (7B) and existing MSLMs of similar model sizes. | Models | Size | VQAv2 | GQA |VizWiz | SQA(IMG) | TextVQA | POPE | MME(P) | MMB |MM-Vet| |:--------:|:-----:|:----:|:----:|:-------------:|:--------:|:-----:|:----:|:-------:|:-------:|:-------:| | [LLaVA-v1.5-lora](https://huggingface.co/liuhaotian/llava-v1.5-7b) | 7B |79.10 | **63.00** |47.80 | 68.40 |58.20| 86.40 | **1476.9** | 66.10 |30.2| | [TinyGPT-V](https://huggingface.co/Tyrannosaurus/TinyGPT-V) | 3B | - | 33.60 | 24.80 | - | - | -| - | - |-| | [LLaVA-Phi](https://github.com/zhuyiche/llava-phi) | 3B | 71.40 | - | 35.90 | 68.40 | 48.60 | 85.00 | 1335.1 | 59.80 |28.9| | [MobileVLM](https://huggingface.co/mtgv/MobileVLM-3B) | 3B | - | 59.00 | - | 61.00 | 47.50 | 84.90 | 1288.9 | 59.60 |-| | [MC-LLaVA-3b](https://huggingface.co/visheratin/MC-LLaVA-3b) | 3B | 64.24 | 49.60 | 24.88 | - | 38.59 | 80.59 | - | - |-| | **Imp-v1 (ours)** | 3B | **79.45** | 58.55 | **50.09** |**69.96**| **59.38** | **88.02**| 1434.0 | **66.49** |**33.1**| ### Examples ![example1](images/example1.png) ## License This project is licensed under the Apache License 2.0 - see the [LICENSE](https://www.apache.org/licenses/LICENSE-2.0) file for details. ## About us This project is maintained by the [MILVLG](https://github.com/MILVLG)@Hangzhou Dianzi University (HDU) led by Prof. Zhou Yu and Jun Yu, and is mainly developed by Zhenwei Shao and Xuecheng Ouyang. We hope our model may serve as a strong baseline to inspire future research on MSLM, as well as its derivative applications on mobile devices and robots. ## Citation If you use our model or refer our work in your studies, please cite: ```bibtex @misc{imp2024, author = {Shao, Zhenwei and Ouyang, Xuecheng and Yu, Zhou and Yu, Jun}, title = {Imp-v1: An Emprical Study of Multimodal Small Language Models}, year = {2024}, url = {https://huggingface.co/MILVLG/imp-v1-3b} } ```