---
title: Index-Translate-9B-Orinfer-E8P-Q4A8
canonical_url: "https://www.modelscope.cn/models/iammiq/Index-Translate-9B-Orinfer-E8P-Q4A8"
md_url: "https://www.modelscope.cn/models/iammiq/Index-Translate-9B-Orinfer-E8P-Q4A8.md"
repository: iammiq/Index-Translate-9B-Orinfer-E8P-Q4A8
last_updated: 2026-10-21
license: apache-2.0
pipeline_tag: image-text-to-text
tasks:
  - image-text-to-text
model_type:
  - qwen3_5
architectures:
  - Qwen3_5ForConditionalGeneration
base_model:
  - IndexTeam/Index-Translate-9B
base_model_relation: quantized
parameters: 2.2B
tensor_type:
  - F32
  - I32
  - F16
  - BF16
  - U32
  - I8
library_name:
  - safetensors
downloads: 6
stars: 0
tags:
  - orinfer
  - jetson
  - e8p
  - int8
  - mixed-precision
  - safetensors
  - translation
---

# Index-Translate-9B-Orinfer-E8P-Q4A8

> Index-Translate-9B-Orinfer-E8P-Q4A8 - iammiq 在 ModelScope 开源的模型。Index-Translate-9B · Orinfer E8P Q4A8

iammiq/Index-Translate-9B-Orinfer-E8P-Q4A8 是 ModelScope 魔搭社区上的 2.2B 参数image-text-to-text模型，采用 apache-2.0 许可，基于 IndexTeam/Index-Translate-9B 构建。

- **Repository**: iammiq/Index-Translate-9B-Orinfer-E8P-Q4A8
- **License**: apache-2.0
- **Tasks**: image-text-to-text
- **Parameters**: 2.2B
- **Base model**: IndexTeam/Index-Translate-9B
- **Tags**: orinfer, jetson, e8p, int8, mixed-precision, safetensors, translation
- **Downloads**: 6
- **Stars**: 0
- **Last updated**: 2026-10-21

Source: https://www.modelscope.cn/models/iammiq/Index-Translate-9B-Orinfer-E8P-Q4A8

---

# Index-Translate-9B · Orinfer E8P Q4A8

[中文](README_zh.md) · [Hugging Face](https://huggingface.co/immiq/Index-Translate-9B-Orinfer-E8P-Q4A8) · [ModelScope](https://modelscope.cn/models/iammiq/Index-Translate-9B-Orinfer-E8P-Q4A8)

A quantized version of [IndexTeam/Index-Translate-9B](https://huggingface.co/IndexTeam/Index-Translate-9B/tree/4e06279e62352dab96bbeb6fa0579bca5e822d75) for **Jetson AGX Orin 64GB / CUDA SM87**, deployed with [Orinfer v0.2.0](https://github.com/iMMIQ/orinfer/releases/tag/v0.2.0). It supports text, images and multiple images, streaming Chat Completions, native MTP, prefix caching, continuous batching and resumable JSONL offline inference. Translation requests use `enable_thinking: false`; thinking follows the original model's chat template.

## Quantization and files

Two-stage residual E8P Q4 with coefficients [4, 1], signed-block128 rotation and beam-8 encoding over the original BF16 weights, paired with INT8 computation. Sensitive tensors retain the validated W8/FP16/BF16/FP32 mixed-precision policy.

Deployed read-only GPU data, including scales, derived constants and precision exceptions: **5,473,766,976 bytes**, approximately **4.536 bits per recorded GPU weight parameter**. Stored safetensors: **5.47 GB** across **6 shards**. Context capacity: **262,144 tokens**; KV cache is INT8 and persistent GDN state is FP32. Quantization seed: `20261002`.

The repository contains HF configuration/tokenizer assets, the native chat template, standard safetensors containers and `cache/model.json`. The descriptor fixes physical tensor layouts and the execution-package digest. The matching Rust `.so`, cubins and their source are distributed through GitHub Release.

## Deploy

Download this model repository into `./model`. Prebuilt deployment uses the aarch64 Rust CLI, NVIDIA CUDA Driver and the matching SM87 execution package.

Execution-package digest: `821345ac103f4aa09ba98d2a09bd8d489d560f1a5ea09f73b396e1f102431efa`.

```bash
curl -fLO https://github.com/iMMIQ/orinfer/releases/download/v0.2.0/orinfer-v0.2.0-linux-aarch64.tar.xz
curl -fLO https://github.com/iMMIQ/orinfer/releases/download/v0.2.0/orinfer-exec-qwen35-9b-a8-sm87-v0.2.0.tar.xz
curl -fLO https://github.com/iMMIQ/orinfer/releases/download/v0.2.0/SHA256SUMS
sha256sum --ignore-missing -c SHA256SUMS
tar -xf orinfer-v0.2.0-linux-aarch64.tar.xz
./orinfer-v0.2.0-linux-aarch64/bin/orinfer package install ./orinfer-exec-qwen35-9b-a8-sm87-v0.2.0.tar.xz
./orinfer-v0.2.0-linux-aarch64/bin/orinfer validate-model ./model
./orinfer-v0.2.0-linux-aarch64/bin/orinfer serve ./model \
  --model index-translate-9b --cuda-graph decode_only --mtp-drafts 3
```

The listener is `[::]:8088`. OpenAI Chat Completions uses `/v1/chat/completions`; set `stream: true` for SSE. Images use `image_url`, and thinking output uses `reasoning_content`. The CLI also provides Ollama-compatible APIs and `orinfer batch` for JSONL jobs. See the [serving guide](https://github.com/iMMIQ/orinfer/blob/v0.2.0/docs/serving.md).

## Measured performance

Jetson AGX Orin 64GB, **2026-10-21**. Throughput is **tok/s**; startup is seconds.

| Prefill 512 | Prefill 8K | Single decode, MTP | 8-request decode | Max requests / decode | Serve → API (s) | Serve → first output (s) |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 2213.52 | 2223.34 | 56.49 | 233.87 | 128 / 890.74 | 2.92 | 3.10 |

Single-request runs enable adaptive MTP up to 3 drafts, with 512/8192 input tokens and 257 fixed output tokens; the single-decode column uses 512-token input. Multi-request runs automatically switch the benchmark to MTP off, with 256 input and 65 output tokens per request. Decode timing starts after the first emitted step, and multi-request timing uses a first-output barrier. Each throughput value is a median of three warm runs after one warmup; fixed-length diagnostics ignore EOS. All use greedy, seed `20261002`, `decode_only`, thinking off and prefix cache off. MTP speed depends on the task and acceptance rate.

Startup uses three fresh Rust `serve` processes and a 21-token translation request: process invocation to `/health` ready, then to the first nonempty SSE output. Existing file caches are preserved; SSE output can contain several tokens. The table reports medians, and file-cache effects plus individual large-model startup observations are in the [performance reference](https://github.com/iMMIQ/orinfer/blob/v0.2.0/docs/performance.md).

## Quality scope

Quantization-baseline diagnostics recorded on 2026-10-18: fixed-history comparison against original BF16, seed `20261002`; mean target NLL delta in nats/token:

| Sample | Positions | Mean NLL delta |
| --- | ---: | ---: |
| Text | 130 | +0.02370 |
| Long input through 8K | 26 | -0.07549 |
| Simple images | 17 | +0.08327 |

The accepted optimized execution has a four-position, two-request comparison against the same quantized single-request reference: maximum absolute NLL delta 0.11037, mean delta 0.02494 and mean KL 0.00197; all selected tokens are in the reference top3. The numerical guard is 0.1, and the remaining batch/cancellation checks from that guarded run remain a separate acceptance item.

These are small-sample diagnostics covering text, inputs through 8K and controlled simple images. Broader task benchmarks, complex image translation and full-context quality remain evaluation areas. Quality and execution-regression scopes are recorded separately in the [performance reference](https://github.com/iMMIQ/orinfer/blob/v0.2.0/docs/performance.md).

## License

Model weights retain [Apache-2.0](LICENSE). Orinfer software and its separate execution package use LGPL-3.0-or-later; source and third-party notices ship with the release.