---
title: Xiaomi-Lens-0-MNN-fp16
canonical_url: "https://www.modelscope.cn/models/yunfeng/Xiaomi-Lens-0-MNN-fp16"
md_url: "https://www.modelscope.cn/models/yunfeng/Xiaomi-Lens-0-MNN-fp16.md"
repository: yunfeng/Xiaomi-Lens-0-MNN-fp16
last_updated: 2026-10-20
license: apache-2.0
pipeline_tag: image-text-to-text
tasks:
- image-text-to-text
base_model:
- SeerRay-Lab/Xiaomi-Lens-0
base_model_relation: finetune
downloads: 0
stars: 0
tags:
- mnn
- xiaomi-lens
- multimodal
- visual-grounding
- image-segmentation
- on-device
- fp16
---
# Xiaomi-Lens-0-MNN-fp16
> Xiaomi-Lens-0-MNN-fp16 - yunfeng 在 ModelScope 开源的模型。Xiaomi-Lens-0-MNN-fp16
yunfeng/Xiaomi-Lens-0-MNN-fp16 是 ModelScope 魔搭社区上的image-text-to-text模型，采用 apache-2.0 许可，基于 SeerRay-Lab/Xiaomi-Lens-0 构建。
- **Repository**: yunfeng/Xiaomi-Lens-0-MNN-fp16
- **License**: apache-2.0
- **Tasks**: image-text-to-text
- **Base model**: SeerRay-Lab/Xiaomi-Lens-0
- **Tags**: mnn, xiaomi-lens, multimodal, visual-grounding, image-segmentation, on-device, fp16
- **Downloads**: 0
- **Stars**: 0
- **Last updated**: 2026-10-20
Source: https://www.modelscope.cn/models/yunfeng/Xiaomi-Lens-0-MNN-fp16
---
# Xiaomi-Lens-0-MNN-fp16
**Xiaomi-Lens-0** (4.6 B vision-language grounding + segmentation model) converted to
[MNN](https://github.com/alibaba/MNN) for on-device CPU inference — no torch, no transformers,
no 9.2 GB bf16 checkpoint. Three MNN graphs plus an fp16 embedding table run the whole
image → answer + bbox + pixel-mask pipeline.
Run it with the `xiaomi-lens-mnn` CLI (downloads ~10.5 GB into
`~/.cache/xiaomi-lens-mnn/fp16` on first use, then runs offline):
```bash
uvx xiaomi-lens-mnn --image cow.jpg \
--prompt 'a large brown cow standing in the grass'
```
```
[xiaomi-lens-mnn] fp16 result
prompt tokens : 255 (216 image, grid 24x36)
completion : 32 tokens
bboxes : [[462, 302, 854, 754]]
masks : 1 (0 in thinking)
timings : vision 1.80s, preprocess 0.04s, prefill 70.11s, decode 6.74s, mask 0.85s, total 79.54s
overlay : lens-out/overlay.png
[462, 302, 854, 754]<|mask|>
```
*586×394 input fitted onto the fixed 576×384 mask canvas. Red is the predicted mask; the
answer text and bbox are identical for HF torch fp32, MNN fp16 and MNN int8. The fp16 mask
matches the reference pixel-for-pixel; int8 differs on 0.03 % of pixels.*
The wheel is published on PyPI as [`xiaomi-lens-mnn`](https://pypi.org/project/xiaomi-lens-mnn/),
so `uvx` resolves it automatically. To pin the tested build:
`uvx --from xiaomi-lens-mnn==0.1.0 xiaomi-lens-mnn --image cow.jpg`.
Quantized alternative: [`Xiaomi-Lens-0-MNN-int8`](https://huggingface.co/yunfengwang/Xiaomi-Lens-0-MNN-int8)
(6.2 GB, ~8 % faster, IoU 0.997 vs fp16 on the test image).
ONNX graphs for re-conversion: [`Xiaomi-Lens-0-onnx`](https://huggingface.co/yunfengwang/Xiaomi-Lens-0-onnx).
## Files
| file | size | precision | role |
| --- | --- | --- | --- |
| `lens_vit.mnn` | 663 MB | fp16 | Qwen3.5 ViT (24 blocks, hidden 1024, patch 16, merge 2); dynamic patch count |
| `lens_mask.mnn` | 167 MB | fp16 | pixel-mask decoder, fixed 24×36 canvas, 32 mask slots |
| `lens_llm_decode.mnn` + `.weight` | 8.42 GB | fp16 | one-token Qwen3.5-4B decoder with explicit KV / conv / recurrent states |
| `embed_tokens_fp16.npy` | 1.27 GB | fp16 | tied input embedding table (248,078 × 2560) |
| `runtime/` | 29 MB | — | config, generation config, processor config, tokenizer, chat template, vision position table |
| `manifest.json` | — | — | per-file size + sha256 |
| `benchmarks.json`, `verification.json` | — | — | speed table and numeric checks below |
| `images/` | 0.6 MB | — | comparison figure and speed chart used in this card |
The tokenizer, the Qwen2-VL smart-resize/patchify, M-RoPE and the vision position/rotary
tables are plain numpy/Pillow in the CLI; only the three graphs run in MNN.
## Speed
Apple M5 Pro (48 GB), macOS arm64, CPU backend, 15 threads. Same 586×394 image, 255-token
prompt (216 image tokens), 32 generated tokens, one mask. Lower is better.
| stage | torch fp32 (HF) | ONNX Runtime | **MNN fp16** | MNN int8 |
| --- | ---: | ---: | ---: | ---: |
| vision | 0.74 s | 1.41 s | **1.80 s** | 1.84 s |
| LLM prefill (255 tok) | 24.78 s | 84.46 s | **70.11 s** | 64.46 s |
| LLM decode (32 tok) | 600.07 s | 10.36 s | **6.74 s** | 6.10 s |
| pixel mask | 0.35 s | 0.15 s | **0.85 s** | 0.53 s |
| end to end | 625.9 s | 96.4 s | **79.5 s** | 73.0 s |
| decode per token | 18.75 s | 0.324 s | **0.211 s** | 0.191 s |
The MNN decoder graph is incremental (one token per session run), so prefill is a 255-step
loop and is slower than torch's parallel prefill; vision and mask are slightly slower than
torch/ONNX too. The decisive win is decode: MNN is ~89× faster per token than the official
HF CPU path, whose GatedDeltaNet `conv1d` falls back to a slow im2col kernel. End to end,
MNN fp16 is **7.9× faster than torch fp32** while using 44 % less disk.
## Verification
Each graph was compared against the official torch/HF implementation before publishing
(details in `verification.json`):
| check | result |
| --- | --- |
| ViT pre-merge features, max / mean abs | 8.4 / 1.3e-3 |
| ViT merged image features, max / mean abs | 7.2e-3 / 3.6e-5 |
| decoder logits / hidden, max abs over 8 steps | 1.8e-3 / 1.5e-3 |
| mask logits, max / mean abs | 6.0e-3 / 2.2e-3 |
| generated text and bbox vs HF | identical |
| final pixel mask vs HF | IoU 1.0000, 100 % pixel agreement |
End-to-end output on the reference image:
`[462, 302, 854, 754]<|mask|>`, foreground ratio 10.05 %.
## Graph I/O
`lens_vit` — inputs `pixel_values (n,1536)` fp16, `pos_embed (n,1024)` fp16, `cos/sin (n,64)`
fp16; outputs `premerge (n,1024)` and `merged (m,2560)` fp16. `n` is dynamic.
`lens_llm_decode` — fixed single token. Inputs `emb (1,1,2560)` fp32, `cos/sin (1,1,64)`
fp32, then per layer `conv{i} (1,8192,3)` + `rec{i} (1,32,128,128)` for the 24 GatedDeltaNet
layers and `k{i}/v{i} (1,4,past,256)` for the 8 full-attention layers (3, 7, …, 31).
Outputs `hidden (1,1,2560)`, `logits (1,1,248078)` and `o_*` states.
`lens_mask` — inputs `mask_hidden (1,32,2560)` fp16, `image_features (1,864,1024)` fp16,
`pad_mask (1,864)` fp32; output `mask_logits (1,32,384,576)` fp32.
## MNN conversion notes
- fp16 ONNX external data is misread by `mnnconvert` 3.6.1, so the decoder ONNX keeps fp32
external data and MNN applies `--fp16`.
- MNN corrupts a single 248,078-wide MatMul (columns past 217,280); the tied `lm_head` is
exported as two 124,039-wide `nn.Linear` MatMuls.
- Split weights must be initializers, not ONNX `Constant` attributes: MNN ignores external
data on constants.
- ViT output tokens are in 2×2 merge-block order; they are permuted to row-major before the
mask decoder.
## Limitations
- The released mask graph is specialized to a **24×36 patch canvas** (576×384).
The CLI fits every image onto that canvas (near-3:2 images are resized
directly, others centre-letterboxed) and maps boxes and masks back to the
original image. Effective input detail is ~0.22 MP; crop first for small objects.
- CPU only; Metal is not enabled in this release.
- One image and one prompt per process. State is reset per `run()`.
- The model may emit masks inside ``; the CLI counts them but overlays only answer masks.
## Run from a local copy
```bash
hf download yunfengwang/Xiaomi-Lens-0-MNN-fp16 --local-dir Xiaomi-Lens-0-MNN-fp16
uvx xiaomi-lens-mnn --image photo.jpg --model-dir Xiaomi-Lens-0-MNN-fp16
```
ModelScope mirror: `modelscope download --model yunfeng/Xiaomi-Lens-0-MNN-fp16 --local_dir Xiaomi-Lens-0-MNN-fp16`
or pass `--source ms` to the CLI.
## License
Apache-2.0. A format conversion of [SeerRay-Lab/Xiaomi-Lens-0](https://huggingface.co/SeerRay-Lab/Xiaomi-Lens-0);
the MNN graphs are derivative weights of that release.