---
title: Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-FP8
canonical_url: "https://www.modelscope.cn/models/chuanSir/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-FP8"
md_url: "https://www.modelscope.cn/models/chuanSir/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-FP8.md"
repository: chuanSir/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-FP8
last_updated: 2026-04-22
license: apache-2.0
pipeline_tag: image-text-to-text
tasks:
  - image-text-to-text
model_type:
  - qwen3_5_moe
architectures:
  - Qwen3_5MoeForConditionalGeneration
base_model:
  - huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated
  - huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated
  - Qwen/Qwen3.6-35B-A3B
base_model_relation: quantized
parameters: 36.0B
tensor_type:
  - BF16
  - F8_E4M3
library_name:
  - transformer
  - safetensors
downloads: 3263
stars: 7
tags:
  - qwen3.6
  - moe
  - vlm
  - fp8
  - quantized
  - blockwise
  - vllm
  - dgx-spark
---

# Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-FP8

> Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-FP8 - chuanSir 在 ModelScope 开源的模型。Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-FP8

chuanSir/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-FP8 是 ModelScope 魔搭社区上的 36.0B 参数image-text-to-text模型，采用 apache-2.0 许可，基于 huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated、huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated、Qwen/Qwen3.6-35B-A3B 构建。

- **Repository**: chuanSir/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-FP8
- **License**: apache-2.0
- **Tasks**: image-text-to-text
- **Parameters**: 36.0B
- **Base model**: huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated, huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated, Qwen/Qwen3.6-35B-A3B
- **Tags**: qwen3.6, moe, vlm, fp8, quantized, blockwise, vllm, dgx-spark
- **Downloads**: 3263
- **Stars**: 7
- **Last updated**: 2026-04-22

Source: https://www.modelscope.cn/models/chuanSir/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-FP8

---

# Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-FP8

**Vision-capable FP8-quantized abliterated Qwen3.6-35B-A3B (MoE, hybrid mamba/attention), built from the Huihui `Claude-4.7-Opus-abliterated` variant. Targets Nvidia DGX Spark and other FP8-capable hardware (~80 GB VRAM for full 262k context).**

## Model Lineage

- Base: [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (BF16, hybrid linear-attention + full-attention MoE with 40 layers, 256 experts, vision encoder)
- Abliterated (refusals removed) by Huihui: [huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated](https://huggingface.co/huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated)
- Claude-4.7-Opus-style tuning by Huihui: [huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated](https://huggingface.co/huihui-ai/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated)
- This repo: FP8 quantization matching the format of [Qwen/Qwen3.6-35B-A3B-FP8](https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8)

## Why FP8

Qwen3.6-35B-A3B in BF16 is ~72 GB on disk. FP8 cuts that to ~37 GB while preserving vision layers and precision-sensitive modules in BF16. The expected throughput uplift on DGX Spark is on par with what we saw for Qwen3.5 (31 → 51 t/s, ~65%).

## Quantization Details

**Scheme**: native FP8 blockwise, identical on-disk format to the official `Qwen/Qwen3.6-35B-A3B-FP8`.

| Field | Value |
|---|---|
| `quant_method` | `fp8` |
| `activation_scheme` | `dynamic` (per-token, at inference) |
| `fmt` | `e4m3` |
| `weight_block_size` | `[128, 128]` |
| Scale dtype / key | `bf16`, `*.weight_scale_inv` |
| Scale shape | `(ceil(out/128), ceil(in/128))` |

**Quantized (weights → FP8 e4m3, per-block `[128, 128]` scales):**
- All 2D Linear `*.weight` in language layers that aren't in the exclusion list, including:
  - `self_attn.{q,k,v,o}_proj` (full-attention layers)
  - `linear_attn.{in_proj_qkv, in_proj_z, out_proj}` (linear-attention / mamba layers)
  - `mlp.shared_expert.{gate,up,down}_proj`
  - **All 256 experts per MoE layer**, un-fused to match Qwen's official per-expert layout:
    - `mlp.experts.{0..255}.{gate_proj, up_proj, down_proj}.weight`

**Kept in BF16** (matches Qwen's `modules_to_not_convert`):

| Module | Reason |
|---|---|
| `lm_head` | Output head — precision-sensitive |
| `model.language_model.embed_tokens` | Embedding layer |
| `*.input_layernorm`, `*.post_attention_layernorm` | LayerNorms |
| `*.self_attn.{q_norm, k_norm}` | QK norms |
| `*.linear_attn.{A_log, conv1d, dt_bias, in_proj_a, in_proj_b, in_proj_ba, norm}` | Mamba state-space params (small, sensitive) |
| `*.mlp.gate`, `*.mlp.shared_expert_gate` | MoE router gates — routing precision matters |
| `model.visual.*` | Entire visual encoder (patch_embed, 27 ViT blocks, deepstack mergers, merger) |
| `mtp.*` | Multi-token prediction module |

## Notable Implementation Notes

- **Source experts were fused 3D** (`mlp.experts.gate_up_proj[256, 1024, 2048]`, `mlp.experts.down_proj[256, 2048, 512]`) — we un-fuse them to the per-expert layout the official Qwen FP8 uses (`mlp.experts.{E}.{gate, up, down}_proj.weight`). This is what vLLM's Fp8 MoE loader expects.
- **Streaming quantization**: processed one source shard at a time on the GPU; peak host memory ~6 GB. Avoids the llmcompressor pitfall where peak VM grew to 168 GB during the Compressing phase and got OOM-killed on the 128 GB DGX Spark unified-memory budget.
- **Sanity check**: round-trip dequantization median relative error **~2.16%** per tensor across sampled MoE expert and attention layers — in line with the E4M3 blockwise profile observed on the sibling repo.
- **Hybrid architecture preserved**: this model has 10 full-attention layers (3, 7, 11, 15, 19, 23, 27, 31, 35, 39) interleaved with 30 linear-attention / mamba layers; both are quantized in the same way but only the full-attention layers have `self_attn.{q,k,v,o}_proj`.

## Numbers

| | BF16 source | This FP8 |
|---|---|---|
| Size on disk | ~72 GB | **37.5 GB** |
| Tensors in index | 1045 (fused experts) | **64189** (un-fused) |
| FP8 weight tensors | — | 31738 |
| BF16 tensors | — | 32451 (incl. 31738 `weight_scale_inv`) |
| Shards | 26 | 26 |

## Loading

```python
from transformers import AutoModelForImageTextToText, AutoProcessor
model = AutoModelForImageTextToText.from_pretrained(
    "batsclamp/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-FP8",
    dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained("batsclamp/Huihui-Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-FP8")
```

For vLLM, point at the repo — the `quantization_config` is already correctly set (`quant_method: fp8`, weight-block `[128, 128]`, dynamic activations).

## Disclaimer

Abliterated model. Not recommended if you expect a polite corporate assistant.