---
title: shot-categorizer-v0
canonical_url: "https://www.modelscope.cn/models/hf-diffusers/shot-categorizer-v0"
md_url: "https://www.modelscope.cn/models/hf-diffusers/shot-categorizer-v0.md"
repository: hf-diffusers/shot-categorizer-v0
last_updated: 2025-03-06
license: mit
model_type:
  - florence2
architectures:
  - Florence2ForConditionalGeneration
base_model:
  - microsoft/Florence-2-large
base_model_relation: finetune
parameters: 829.2M
tensor_type:
  - F16
library_name:
  - pytorch
  - transformer
  - safetensors
frameworks:
  - pytorch
language:
  - en
inference_backends:
  - "deploy_task vlm/text/emb"
  - "vllm 0.9.2"
downloads: 615
stars: 0
---

# shot-categorizer-v0

> shot-categorizer-v0 - hf-diffusers 在 ModelScope 开源的模型。Shot categorization model finetuned from the microsoft/Florence-2-large model. This model can be used to obtain metadata information about shots which can further be used to curate datasets of different…

hf-diffusers/shot-categorizer-v0 是 ModelScope 魔搭社区上的 829.2M 参数机器学习模型，采用 mit 许可，基于 microsoft/Florence-2-large 构建，可用 deploy_task vlm/text/emb、vllm 0.9.2 部署。

- **Repository**: hf-diffusers/shot-categorizer-v0
- **License**: mit
- **Parameters**: 829.2M
- **Base model**: microsoft/Florence-2-large
- **Inference backends**: deploy_task vlm/text/emb, vllm 0.9.2
- **Downloads**: 615
- **Stars**: 0
- **Last updated**: 2025-03-06

Source: https://www.modelscope.cn/models/hf-diffusers/shot-categorizer-v0

---

# Shot Categorizer 🎬

<div align="center">
  <img src="assets/header.jpg"/>
</div>

Shot categorization model finetuned from the [`microsoft/Florence-2-large`](https://huggingface.co/microsoft/Florence-2-large) model. This
model can be used to obtain metadata information about shots which can further be used to curate datasets of different kinds.

Training configuration:

* Batch size: 16
* Gradient accumulation steps: 4
* Learning rate: 1e-6
* Epochs: 20
* Max grad norm: 1.0
* Hardware: 8xH100s

Training was conducted using FP16 mixed-precision and DeepSpeed Zero2 scheme. The vision tower of the model
was kept frozen during the training. We used the [diffusers/ShotDEAD-v0](https://huggingface.co/datasets/diffusers/ShotDEAD-v0)
dataset for conducting training. 

Training code is available [here](https://github.com/huggingface/movie-shot-categorizer).

## Inference

```py
from transformers import AutoModelForCausalLM, AutoProcessor
import torch
from PIL import Image
import requests


folder_path = "diffusers/shot-categorizer-v0"
model = (
    AutoModelForCausalLM.from_pretrained(folder_path, torch_dtype=torch.float16, trust_remote_code=True)
    .to("cuda")
    .eval()
)
processor = AutoProcessor.from_pretrained(folder_path, trust_remote_code=True)

prompts = ["<COLOR>", "<LIGHTING>", "<LIGHTING_TYPE>", "<COMPOSITION>"]
img_path = "./assets/image_3.jpg"
image = Image.open(img_path).convert("RGB")

with torch.no_grad() and torch.inference_mode():
    for prompt in prompts:
        inputs = processor(text=prompt, images=image, return_tensors="pt").to("cuda", torch.float16)
        generated_ids = model.generate(
            input_ids=inputs["input_ids"],
            pixel_values=inputs["pixel_values"],
            max_new_tokens=1024,
            early_stopping=False,
            do_sample=False,
            num_beams=3,
        )
        generated_text = processor.batch_decode(generated_ids, skip_special_tokens=False)[0]
        parsed_answer = processor.post_process_generation(
            generated_text, task=prompt, image_size=(image.width, image.height)
        )
        print(parsed_answer)

```

Should print:

```bash
{'<COLOR>': 'Cool, Saturated, Cyan, Blue'}
{'<LIGHTING>': 'Soft light, Low contrast'}
{'<LIGHTING_TYPE>': 'Daylight, Sunny'}
{'<COMPOSITION>': 'Left heavy'}
```