---
title: OmniCoder-2-9B
canonical_url: "https://www.modelscope.cn/models/Tesslate/OmniCoder-2-9B"
md_url: "https://www.modelscope.cn/models/Tesslate/OmniCoder-2-9B.md"
repository: Tesslate/OmniCoder-2-9B
last_updated: 2026-03-24
license: apache-2.0
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - qwen3_5
architectures:
  - Qwen3_5ForConditionalGeneration
base_model:
  - Qwen/Qwen3.5-9B
base_model_relation: finetune
parameters: 9.4B
tensor_type:
  - BF16
library_name:
  - transformer
  - safetensors
  - pytorch
frameworks:
  - pytorch
language:
  - en
downloads: 1146
stars: 6
tags:
  - qwen3.5
  - code
  - agent
  - sft
  - omnicoder
  - tesslate
---

# OmniCoder-2-9B

> OmniCoder-2-9B - Tesslate 在 ModelScope 开源的模型。The second generation of OmniCoder. Faster reasoning, dramatically less repetition, completely rebuilt training pipeline.

Tesslate/OmniCoder-2-9B 是 ModelScope 魔搭社区上的 9.4B 参数text-generation模型，采用 apache-2.0 许可，基于 Qwen/Qwen3.5-9B 构建。

- **Repository**: Tesslate/OmniCoder-2-9B
- **License**: apache-2.0
- **Tasks**: text-generation
- **Parameters**: 9.4B
- **Base model**: Qwen/Qwen3.5-9B
- **Tags**: qwen3.5, code, agent, sft, omnicoder, tesslate
- **Downloads**: 1146
- **Stars**: 6
- **Last updated**: 2026-03-24

Source: https://www.modelscope.cn/models/Tesslate/OmniCoder-2-9B

---

<div align="center">

<img src="omnicoder-banner.png" alt="OmniCoder 2" width="720">

# OmniCoder-2-9B

### The second generation of OmniCoder. Faster reasoning, dramatically less repetition, completely rebuilt training pipeline.

[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Base Model](https://img.shields.io/badge/Base-Qwen3.5--9B-purple)](https://huggingface.co/Qwen/Qwen3.5-9B)
[![Previous](https://img.shields.io/badge/v1-OmniCoder--9B-gray)](https://huggingface.co/Tesslate/OmniCoder-9B)

---

</div>

## What's New in v2

OmniCoder-2-9B is a ground-up rebuild of [OmniCoder-9B](https://huggingface.co/Tesslate/OmniCoder-9B), not a continuation. The entire training pipeline was redesigned to fix the core issues users reported with v1:

- **No more repetition loops.** v1 trained on ALL tokens (system prompts, tool outputs, templates), which taught the model to reproduce repetitive boilerplate. v2 trains **only on assistant tokens**. The model never learns to parrot templates, so it stays coherent through long multi-turn conversations.
- **Faster, more focused reasoning.** v1's `<think>` blocks were often bloated and circular. v2 produces tighter reasoning chains that get to the point faster because it only learned from the actual reasoning, not the scaffolding around it.
- **Much more stable in agentic loops.** v1 would sometimes get stuck in repetitive tool-call cycles. v2 handles multi-step agentic tasks cleanly. It knows when to stop, when to switch tools, and when to give a final answer.
- **Rebuilt training pipeline.** Switched from all-token cosine-schedule training to assistant-only constant-LR training (based on Schulman's "LoRA Without Regret"). The model converges faster and doesn't suffer from premature LR decay that killed v1's learning.

> **TL;DR:** OmniCoder-2-9B fixes the repetition and instability issues from v1. Same weights, same architecture, completely different training approach.

---

## Benchmarks

<div align="center">

| Benchmark | **OmniCoder-2-9B** | OmniCoder-9B | Qwen3.5-9B | GPT-OSS-120B | GLM 4.7 | Claude Haiku 4.5 |
|:---|:---:|:---:|:---:|:---:|:---:|:---:|
| **AIME 2025** (pass@5) | **90** | 90 | 91.6 | | | |
| **GPQA Diamond** (pass@1) | **83** | 83.8 | 81.7 | 71.5 | | 73 |
| **GPQA Diamond** (pass@3) | **86** | 86.4 | | | | |
| **Terminal-Bench 2.0** | **25.8** | 23.6 | 14.6 | | 33.4 | 27 |

</div>

### Key Results

- **GPQA Diamond pass@1: 83%** (164/198). On par with OmniCoder-9B (83.8%) and Qwen3.5-9B base (81.7%). Pass@3 improves to 86%.
- **Terminal-Bench 2.0: 25.8%** (23/89). +2.2 points over OmniCoder-9B (23.6%), +76% over Qwen3.5-9B base (14.6%).
- **AIME 2025 pass@5: 90%** (27/30). Parity with OmniCoder-9B.

---

## Quickstart

### Transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Tesslate/OmniCoder-2-9B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

messages = [
    {"role": "system", "content": "You are a helpful coding assistant."},
    {"role": "user", "content": "Write a Python function to find the longest common subsequence of two strings."},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=2048, temperature=0.6, top_p=0.95, top_k=20)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[-1]:], skip_special_tokens=True))
```

### vLLM

```bash
vllm serve Tesslate/OmniCoder-2-9B --tensor-parallel-size 1 --max-model-len 65536
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="token")
response = client.chat.completions.create(
    model="Tesslate/OmniCoder-2-9B",
    messages=[{"role": "user", "content": "Explain the difference between a mutex and a semaphore."}],
    temperature=0.6,
)
print(response.choices[0].message.content)
```

### llama.cpp (GGUF)

```bash
llama-cli --hf-repo Tesslate/OmniCoder-2-9B-GGUF --hf-file omnicoder-2-9b-q4_k_m.gguf -p "Your prompt" -c 8192
```

---

## Training Details

### v1 vs v2 Pipeline

| | OmniCoder-9B (v1) | **OmniCoder-2-9B (v2)** |
|:---|:---|:---|
| **Loss computed on** | All tokens (system, user, tool, assistant) | **Assistant tokens only** |
| **LR schedule** | Cosine (decayed to zero too early) | **Constant with warmup** |
| **EOS training** | Global | **Per-turn** (`train_on_eos: turn`) |
| **Thinking blocks** | Stripped by stock template | **Preserved on all turns** (custom Jinja2) |
| **Repetition** | Frequent loops in generation | **Near-zero** |
| **Convergence** | LR killed learning at step 60 | **Clean convergence at step 80-100** |

### Training Config

| | |
|:---|:---|
| **Base Model** | [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) |
| **Method** | LoRA SFT (r=64, alpha=32, all layers incl. MLP) |
| **Dataset** | 425K agentic trajectories from 5 sources |
| **Loss** | Assistant tokens only (`roles_to_train: [assistant]`) |
| **Sequence Length** | 65,536 tokens (sample packing) |
| **LR Schedule** | Constant with warmup (2e-4, 10 warmup steps) |
| **Hardware** | 4x NVIDIA H200 (DDP) |
| **Framework** | Axolotl |
| **Precision** | bf16 |
| **Optimizer** | AdamW (weight_decay=0.001) |
| **Effective Batch** | 32 |
| **Steps** | 350 (~10% of one epoch) |

### Training Data Sources

| Source | Samples | Description |
|:---|---:|:---|
| NVIDIA Nemotron-Terminal-Corpus | 226K | Terminal agent trajectories |
| CoderForge-Preview (reward >= 0.5) | 155K | SWE-bench style coding trajectories |
| Nemotron Skill-Based | 24K | Skill-based coding tasks |
| Scale-SWE | 20K | Real GitHub issue patches (synthesized trajectories) |
| Opus Reasoning | 2.3K | Chain-of-thought reasoning |

### Why Constant LR? (Schulman "LoRA Without Regret")

v1 used a cosine LR schedule over 80 steps. The learning rate decayed to zero by step 60, **killing learning before the model converged**. Loss appeared to plateau at 0.45 but was actually still dropping.

v2 follows Schulman et al.'s findings:
- **Constant LR.** No decay, no premature convergence death.
- **LoRA LR ~10x FullFT.** Our 2e-4 is correct for 9B-class models.
- **LoRA on ALL layers including MLP.** Attention-only LoRA significantly underperforms.
- **Batch size 32.** LoRA is less tolerant of large batches; 32 is the sweet spot.

---

## Architecture

OmniCoder-2 inherits Qwen3.5-9B's hybrid architecture:

- **Gated Delta Networks.** Linear attention layers interleaved with standard attention for efficient long-range dependencies.
- **VLM Backbone.** Built on `Qwen3_5ForConditionalGeneration`.
- **262K Native Context.** Full 262,144 token context window.

---

## Recommended Sampling Parameters

| Parameter | Value |
|:---|:---|
| Temperature | 0.6 |
| Top-P | 0.95 |
| Top-K | 20 |
| Presence Penalty | 0.0 |

For agentic / tool-calling tasks, consider lower temperature (0.2-0.4) for more deterministic behavior.

---

## Limitations

- Performance on non-English tasks has not been extensively evaluated
- Tool-calling format is flexible but works best with the scaffolding patterns seen in training

---

## Acknowledgments

Special thanks to the [Axolotl](https://github.com/axolotl-ai-cloud/axolotl) team and the discussion in [axolotl#3453](https://github.com/axolotl-ai-cloud/axolotl/issues/3453) for helping get Qwen3.5 packing support working.

---

## Citation

```bibtex
@misc{omnicoder2_2025,
  title={OmniCoder-2-9B: A Frontier Open Coding Agent},
  author={Tesslate},
  year={2025},
  url={https://huggingface.co/Tesslate/OmniCoder-2-9B}
}
```

---

<div align="center">

**Built by [Tesslate](https://tesslate.com)**

</div>