---
title: Qwen3.5-0.8B-GDN2
canonical_url: "https://www.modelscope.cn/models/tysgydk/Qwen3.5-0.8B-GDN2"
md_url: "https://www.modelscope.cn/models/tysgydk/Qwen3.5-0.8B-GDN2.md"
repository: tysgydk/Qwen3.5-0.8B-GDN2
last_updated: 2026-09-19
license: apache-2.0
pipeline_tag: image-text-to-text
tasks:
  - image-text-to-text
model_type:
  - qwen3_5_gdn2
architectures:
  - Qwen35GDN2ForConditionalGeneration
base_model:
  - Qwen/Qwen3.5-0.8B
base_model_relation: finetune
parameters: 965.7M
tensor_type:
  - BF16
library_name:
  - pytorch
  - transformer
  - safetensors
frameworks:
  - Pytorch
downloads: 29
stars: 0
tags:
  - gated-delta-net
  - gdn2
  - linear-attention
  - multimodal
---

# Qwen3.5-0.8B-GDN2

> Qwen3.5-0.8B-GDN2 - tysgydk 在 ModelScope 开源的模型。一个社区基线模型：将千问 3.5（0.8B）全部 18 层 GDN（门控 delta 网络）线性注意力 改造为 GDN2（逐通道擦除/写入门控），完整保留多模态能力（视觉塔、M-RoPE、 对话模板）。

tysgydk/Qwen3.5-0.8B-GDN2 是 ModelScope 魔搭社区上的 965.7M 参数image-text-to-text模型，采用 apache-2.0 许可，基于 Qwen/Qwen3.5-0.8B 构建。

- **Repository**: tysgydk/Qwen3.5-0.8B-GDN2
- **License**: apache-2.0
- **Tasks**: image-text-to-text
- **Parameters**: 965.7M
- **Base model**: Qwen/Qwen3.5-0.8B
- **Tags**: gated-delta-net, gdn2, linear-attention, multimodal
- **Downloads**: 29
- **Stars**: 0
- **Last updated**: 2026-09-19

Source: https://www.modelscope.cn/models/tysgydk/Qwen3.5-0.8B-GDN2

---

# Qwen3.5-0.8B-GDN2

一个社区基线模型：**将千问 3.5（0.8B）全部 18 层 GDN（门控 delta 网络）线性注意力
改造为 GDN2**（逐通道擦除/写入门控），完整保留多模态能力（视觉塔、M-RoPE、
对话模板）。

A community baseline: **Qwen3.5-0.8B with all 18 GDN (gated delta net) linear-attention
layers converted to GDN2** (per-channel erase/write gates), keeping the full multimodal
stack (vision tower, M-RoPE, chat template) intact.

改造从一个**保函数的权重映射**开始（把逐头的门控平铺为逐通道），因此改造后的模型
在初始时与原模型逐位可比（评测 loss 差 < 0.001），随后在 50% 语言回放的数据配比下
做继续训练，防止灾难性遗忘。

The conversion starts from a **function-preserving weight mapping** (per-head gates tiled
to per-channel), so the converted model is bit-comparable to the original at init
(eval-loss delta < 0.001), then is adapted with a full-parameter continued-training run
under a 50% language-replay mix to prevent forgetting.

## 与原版的区别 | What's different from the original

| 组件 Component | 原版千问 3.5 Original Qwen3.5 | 本模型 This model |
|---|---|---|
| 线性注意力 Linear attention | GDN（逐头标量门控）/ GDN (per-head scalar gates) | GDN2（逐通道擦除/写入门控）/ GDN2 (per-channel erase/write gates) |
| 全注意力 Full attention | 6 层（未动）/ 6 layers (untouched) | 6 层（同权重，经再适配）/ 6 layers (same weights, re-adapted) |
| 视觉塔 Vision tower | Qwen3.5 ViT | 冻结，完全一致 / frozen, identical |
| 视觉通路 Vision path | 直接融合 / direct merge | 直接融合（曾训练一个残差视觉 adapter，消融实验证明它导致 TextVQA 掉 11 个点，已移除）/ direct merge (a residual vision adapter was trained but removed after ablation showed it cost 11 TextVQA points) |
| MTP 头 MTP head | 有 / present | 保留，未使用 / kept, unused |

## 使用方法 | Usage

```python
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained(
    "tysgydk/Qwen3.5-0.8B-GDN2", trust_remote_code=True,
    dtype=torch.bfloat16, device_map="cuda")
processor = AutoProcessor.from_pretrained("tysgydk/Qwen3.5-0.8B-GDN2", trust_remote_code=True)
```

环境要求：`transformers >= 5.17`、`flash-linear-attention >= 0.5.2`
（`pip install flash-linear-attention`）。

Requirements: `transformers >= 5.17`, `flash-linear-attention >= 0.5.2`
(`pip install flash-linear-attention`).

默认 `use_cache=True` 的生成已支持（通过 FLA 缓存桥，把卷积态和循环态桥接进
HF 的 cache 对象）。

Generation with the default `use_cache=True` is supported via an FLA cache bridge
(conv + recurrent states bridged into the HF cache object).

## 训练配方 | Training recipe

- A 阶段：在合成属性数据上训练网格感知的近恒等视觉压缩器（已否决：随机槽位注意力
  会破坏形状/颜色/位置绑定）。
- Stage A: grid-aware near-identity vision compressor on synthetic attribute data (rejected:
  random slot attention — destroys shape/color/position binding).
- B 阶段：先只调 GDN2 门控，再全量解冻 GDN2；50% 语言回放
  （ShareGPT/COIG-CQIA/UltraChat/GSM8K/NuminaMath/哲学）+ 50% 多模态。
- Stage B: GDN2 gate-only adaptation, then full GDN2 unfreeze; 50% language replay
  (ShareGPT/COIG-CQIA/UltraChat/GSM8K/NuminaMath/philosophy) + 50% multimodal.
- C 阶段：在 ChartQA（2.83 万）+ 合成属性混合数据上重训视觉 adapter。
- Stage C: vision adapter retrained on ChartQA (28.3k) + synthetic attr mix.
- 发布训练：全参数（整个语言模型）继续训练 5 万步，lr 1e-5 余弦退火，
  50% 语言回放 / 50% 多模态（ChartQA + 合成）。
- Release run: full-parameter (LM-wide) continued training, 50k steps, lr 1e-5 cosine,
  50% language replay / 50% multimodal (ChartQA + synthetic).
- 事后消融：C 阶段的视觉 adapter 被证明是照片域 VQA 回归的唯一来源（同协议下
  不带 adapter 63.3% vs 带 adapter 52.0%，TextVQA val 512）。用真实照片问答
  做了 3000 步针对性抛光也救不回来，所以发布版**不带** adapter（与原版一致的
  直接视觉融合）。
- Post-hoc ablation: the Stage-C vision adapter turned out to be the sole source of the
  photo-VQA regression (gates-only 63.3% vs with-adapter 52.0% on TextVQA val 512,
  same protocol). A targeted 3k-step polish on real-photo QA did not recover it, so the
  release ships **without** the adapter (direct vision merge, as in the original).

## 评测 | Evaluation

全部为同协议头对头实测（TextVQA：val 512 张，原生分辨率）：

All numbers same-protocol, side-by-side (TextVQA: val 512, native resolution):

| 基准 Benchmark | 原版千问 3.5-0.8B Original Qwen3.5-0.8B | GDN2（本模型）GDN2 (this) |
|---|---|---|
| MMLU 5-shot（n=1000）/ MMLU 5-shot (n=1000) | 42.8% | **45.2%** |
| TextVQA-lite val 512（精确/宽松）/ TextVQA-lite val 512 (exact / soft) | 63.1% / 72.9% | **63.3% / 73.6%** |
| 语言评测 loss（96 条留出集）/ Language eval loss (96-sample holdout) | 2.067 | **1.776** |
| 合成属性任务（n=40）/ Synthetic attribute task (n=40) | — | **39/40** |

发布说明：发布的权重是把 5 万步训出的 GDN2 层**嫁接**到冻结的原版
注意力/MLP/embedding 主干上。另有一个全参数 5 万步变体，语言 loss 略好（1.771），
但照片域短答案的终止符行为崩坏（TextVQA 精确率 23%），因此嫁接版作为发布默认。

Release note: the shipped weights graft the 50k-step-trained GDN2 layers onto the frozen
original attention/MLP/embedding backbone. A full-parameter 50k variant scored slightly
better on language loss (1.771) but regressed short-answer termination on photos
(TextVQA exact 23%), so the graft is the release default.

## 已知限制 | Known limitations

- 0.8B 研究型基线，未做生产级对话对齐。
- 0.8B research baseline; not aligned for production chat use.
- 最终评测在原生图像分辨率下进行；部分训练阶段图像上限约 1024 个融合 token。
- Final evals run at native image resolution; some training stages capped images at
  ~1024 merged tokens.
- MTP 头保留但自定义建模代码未使用。
- MTP head is carried over but unused by the custom modeling code.
- 不强制要求 `causal_conv1d`（缺失时回退到较慢的参考实现）。
- `causal_conv1d` not required (falls back to a slower reference path).

## 引用 | Citation

基于千问 3.5（Apache-2.0）和 FLA 的 GatedDeltaNet2 构建。使用本权重请引用
千问与 FLA/GDN 的原始论文。

Built on Qwen3.5 (Apache-2.0) and FLA's GatedDeltaNet2. Please cite the original
Qwen and FLA/GDN papers if you use this checkpoint.