---
title: Qwen3-VL-Reranker-8B-AWQ
canonical_url: "https://www.modelscope.cn/models/yobo2u/Qwen3-VL-Reranker-8B-AWQ"
md_url: "https://www.modelscope.cn/models/yobo2u/Qwen3-VL-Reranker-8B-AWQ.md"
repository: yobo2u/Qwen3-VL-Reranker-8B-AWQ
last_updated: 2026-05-18
license: apache-2.0
model_type:
  - qwen3_vl
architectures:
  - Qwen3VLForConditionalGeneration
base_model:
  - Qwen/Qwen3-VL-Reranker-8B
base_model_relation: quantized
parameters: 8.8B
tensor_type:
  - I32
  - F16
library_name:
  - safetensors
downloads: 303
stars: 0
tags:
  - qwen
  - qwen3-vl
  - reranker
  - awq
  - quantization
  - 4bit
  - multimodal
---

# Qwen3-VL-Reranker-8B-AWQ

> Qwen3-VL-Reranker-8B-AWQ - yobo2u 在 ModelScope 开源的模型。Qwen3-VL-Reranker-8B-AWQ

yobo2u/Qwen3-VL-Reranker-8B-AWQ 是 ModelScope 魔搭社区上的 8.8B 参数机器学习模型，采用 apache-2.0 许可，基于 Qwen/Qwen3-VL-Reranker-8B 构建。

- **Repository**: yobo2u/Qwen3-VL-Reranker-8B-AWQ
- **License**: apache-2.0
- **Parameters**: 8.8B
- **Base model**: Qwen/Qwen3-VL-Reranker-8B
- **Tags**: qwen, qwen3-vl, reranker, awq, quantization, 4bit, multimodal
- **Downloads**: 303
- **Stars**: 0
- **Last updated**: 2026-05-18

Source: https://www.modelscope.cn/models/yobo2u/Qwen3-VL-Reranker-8B-AWQ

---

# Qwen3-VL-Reranker-8B-AWQ

This repository contains a **language-only AWQ 4-bit quantized** version of [Qwen/Qwen3-VL-Reranker-8B](https://www.modelscope.cn/models/Qwen/Qwen3-VL-Reranker-8B).

The quantization target is practical deployment on constrained GPUs such as **NVIDIA A10 24GB**, especially when colocating the reranker model with the matching embedding model on the same card.

## Quantization summary

- Base model: `Qwen/Qwen3-VL-Reranker-8B`
- Quantization: AWQ 4-bit
- Quantized modules: `model.language_model.layers.*` linear layers
- Preserved in fp16/bf16: visual tower, visual projector/merger, embeddings, normalization layers, `lm_head`
- AWQ config:

```json
{
  "bits": 4,
  "group_size": 128,
  "modules_to_not_convert": ["visual", "lm_head"],
  "quant_method": "awq",
  "version": "gemm",
  "zero_point": true
}
```

## Size and loading footprint

- Quantized repository size: about **6.8GB**
- Observed GPU allocation in smoke loading test: about **6.77GiB**

This makes it feasible to test this reranker together with the matching AWQ embedding model on a single 24GB GPU, with conservative context length and batch/concurrency settings.

## Validation

A small text-only reranking smoke test was run against the original fp16/bf16 checkpoint.

Results:

```text
num_pairs: 12
groups: 3 query groups x 4 docs
top1_match_rate: 1.0
Spearman per group: 0.8 / 0.8 / 0.8
```

The top-1 result matched for all test query groups. Some non-top1 internal ordering changed slightly, which is expected for a small AWQ smoke test.

Validation artifact:

```text
reranker_awq_consistency_text_only.json
```

Note: this is a small text-only smoke test, not a full multimodal reranking benchmark. The visual tower is intentionally kept unquantized to reduce multimodal quality loss.

## Loading example

This model was produced with a custom Qwen3-VL language-only AutoAWQ wrapper. The quantization script is included in this repository:

```text
quant_qwen3vl_awq_language_only.py
```

Example:

```python
import sys
sys.path.insert(0, ".")
from quant_qwen3vl_awq_language_only import register_qwen3vl_awq
register_qwen3vl_awq()

from awq import AutoAWQForCausalLM

model = AutoAWQForCausalLM.from_quantized(
    "yobo2u/Qwen3-VL-Reranker-8B-AWQ",
    trust_remote_code=True,
    safetensors=True,
    device_map={"": 0},
    fuse_layers=False,
)
```

## Suggested A10 24GB deployment starting point

When colocating this reranker model with the matching embedding model on one A10 24GB GPU, start conservatively:

```text
max_model_len: 2048-4096
max_num_seqs: 1
max_num_batched_tokens: 2048-4096
limit_mm_per_prompt: {"image": 1}
```

If serving two separate processes on one GPU, start around:

```text
Embedding gpu_memory_utilization: 0.40-0.45
Reranker  gpu_memory_utilization: 0.40-0.45
```

Then increase context/concurrency empirically.

## License

This derivative quantized model follows the original model license: **Apache-2.0**.

## ModelScope remote snapshot test

This repository version was downloaded back from ModelScope with `snapshot_download` and tested locally after upload.

Test metadata:

```text
repo_id: yobo2u/Qwen3-VL-Reranker-8B-AWQ
source: ModelScope snapshot_download
test_time: 2026-05-18 09:31:44 CST
max_length: 512
query_groups: 4
query_doc_pairs: 16
```

Load / memory:

```text
load_ok: true
load_seconds: 7.647
gpu_alloc_gib_after_load: 6.761
gpu_reserved_gib_after_load: 6.812
```

Reranking smoke result:

```text
top1_expected_relevant_match_count: 4 / 4
top1_expected_relevant_all_match: true
latency_seconds_min: 0.0529
latency_seconds_mean: 0.1646
latency_seconds_max: 1.5346
```

The remote ModelScope snapshot loaded successfully with the included `quant_qwen3vl_awq_language_only.py` registration helper and passed a small 4-query / 16-pair text-only reranking smoke test. This is a functional smoke test, not a full retrieval benchmark.

Full JSON result:

```text
modelscope_remote_reranker_awq_test_results.json
```