--- title: Ternary-Bonsai-2-2B-gguf canonical_url: "https://www.modelscope.cn/models/prism-ml/Ternary-Bonsai-2-2B-gguf" md_url: "https://www.modelscope.cn/models/prism-ml/Ternary-Bonsai-2-2B-gguf.md" repository: prism-ml/Ternary-Bonsai-2-2B-gguf last_updated: 2026-11-05 license: apache-2.0 pipeline_tag: image-text-to-text tasks: - image-text-to-text base_model: - Qwen/Qwen3.5-2B base_model_relation: quantized library_name: - gguf - pytorch frameworks: - pytorch downloads: 472 stars: 1 tags: - ternary - 2-bit - gguf - llama-cpp - cuda - metal - on-device - hybrid-attention - vision - prismml - bonsai --- # Ternary-Bonsai-2-2B-gguf > Ternary-Bonsai-2-2B-gguf - prism-ml 在 ModelScope 开源的模型。Prism ML Website  |  Demo & Examples  |  Discord prism-ml/Ternary-Bonsai-2-2B-gguf 是 ModelScope 魔搭社区上的image-text-to-text模型，采用 apache-2.0 许可，基于 Qwen/Qwen3.5-2B 构建。 - **Repository**: prism-ml/Ternary-Bonsai-2-2B-gguf - **License**: apache-2.0 - **Tasks**: image-text-to-text - **Base model**: Qwen/Qwen3.5-2B - **Tags**: ternary, 2-bit, gguf, llama-cpp, cuda, metal, on-device, hybrid-attention, vision, prismml, bonsai - **Downloads**: 472 - **Stars**: 1 - **Last updated**: 2026-11-05 Source: https://www.modelscope.cn/models/prism-ml/Ternary-Bonsai-2-2B-gguf ---

Prism ML Website  |  Demo & Examples  |  Discord

# Bonsai 2 Ternary 2B: GGUF A 2B-class vision-language model with ternary transformer weights, for llama.cpp (CUDA, Metal, CPU). > **~0.52 GB** language model (PQ2_0), down from ~3.8 GB at FP16 ## Highlights - **Ternary language weights** across embeddings, attention projections, MLP projections and the (tied) LM head, stored with a group-wise FP16 scale. - **Three GGUF packings** of the same weights, consumed directly by custom kernels and never expanded back to FP16: **PTQ1_0** (dense trits, 0.43 GB), **PQ2_0** (2-bit slots, 0.52 GB) and **Q2_0** (0.55 GB). - **Vision input** through a separate mmproj pack (Q8_0, or BF16 for reference precision), loaded only when an image arrives. - **262K-token context**, inherited from the Qwen3.5-2B hybrid-attention backbone (18 linear-attention and 6 full-attention layers). ## Model Overview | Item | Specification | | :--- | :--- | | Base model | Derived from Qwen3.5-2B (architecture unchanged) | | Parameters | ~1.88B language model (24 blocks, hidden 2048, tied embedding/LM head) + ~0.33B vision tower (24 blocks) | | Architecture | Hybrid attention (3 linear-attention : 1 full-attention), SwiGLU MLP, RoPE, RMSNorm | | Context length | 262,144 tokens (inherited from the base model) | | Weight format | Ternary {−1, 0, +1} with FP16 group-wise scales, packed as PTQ1_0, PQ2_0 or Q2_0 | | Weight basis | Blockwise Hadamard rotation folded into the stored weights; the runtime applies the matching transform to activations | | Higher-precision tensors | Normalization weights and the linear-attention state path (`ssm_alpha`, `ssm_beta`) stay F32 | | Vision tower | Optional mmproj pack: Q8_0 (0.36 GB, recommended) or BF16 (0.67 GB) | | Backends | llama.cpp (CUDA, Metal, CPU): every file runs on the PrismML fork; `Q2_0` also runs on upstream llama.cpp [b11556](https://github.com/ggml-org/llama.cpp/releases/tag/b11556) or newer | | License | Apache 2.0 | ## Files | File | Contents | Size | Bits/weight (language model) | | :--- | :--- | ---: | ---: | | `Ternary-Bonsai-2-2B-PTQ1_0.gguf` | language model, dense trits | 0.43 GB | ~1.83 | | `Ternary-Bonsai-2-2B-PQ2_0.gguf` | language model, 2-bit slots | 0.52 GB | ~2.20 | | `Ternary-Bonsai-2-2B-Q2_0.gguf` | language model, Q2_0 | 0.55 GB | ~2.33 | | `Ternary-Bonsai-2-2B-mmproj-Q8_0.gguf` | vision tower | 0.36 GB | | | `Ternary-Bonsai-2-2B-mmproj-BF16.gguf` | vision tower (reference precision) | 0.67 GB | | Bits/weight is file size over the ~1.88B language-model parameters, so it includes the F32 tensors and metadata. The three language-model packs hold the same ternary weights; they differ only in storage layout and kernel speed. ## Quickstart > **[PrismML-Eng/Bonsai-demo](https://github.com/PrismML-Eng/Bonsai-demo) is the source of truth for running these models.** It has the tested setup for every backend and pins a known-good binary release. ### Use the PrismML llama.cpp build The ternary kernels live in the [PrismML-Eng/llama.cpp](https://github.com/PrismML-Eng/llama.cpp) fork (branch `prism`). It runs every file here. Upstream llama.cpp runs the `Q2_0` file from release [b11556](https://github.com/ggml-org/llama.cpp/releases/tag/b11556) onward (see *Use upstream llama.cpp* below). It doesn't have the `PTQ1_0` and `PQ2_0` types yet, so it rejects those files as unknown types. ```bash # prebuilt: pick the archive for your platform # https://github.com/PrismML-Eng/llama.cpp/releases/latest tar -xzf llama--bin-.tar.gz -C bin --strip-components=1 # or build it git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp cmake -B build -DGGML_CUDA=ON && cmake --build build -j # drop -DGGML_CUDA=ON on macOS; Metal is the default ``` ### Text ```bash ./bin/llama-cli -m Ternary-Bonsai-2-2B-PQ2_0.gguf -ngl 99 -fa on \ -p "Explain in three sentences why the sky is blue." ``` ### Image + text ```bash ./bin/llama-mtmd-cli -m Ternary-Bonsai-2-2B-PQ2_0.gguf \ --mmproj Ternary-Bonsai-2-2B-mmproj-Q8_0.gguf -ngl 99 \ --image photo.jpg -p "Describe this image." ``` **Vision projector:** use the Q8_0 mmproj. With the current fork build, the BF16 mmproj can crash llama-server when several image requests run at once; Q8_0 is not affected and quality is nearly the same. **Small text in images:** for small or dense text, raising the image token budget with `--image-min-tokens 1024` may help. ### OpenAI-compatible server ```bash ./bin/llama-server -m Ternary-Bonsai-2-2B-PQ2_0.gguf \ --mmproj Ternary-Bonsai-2-2B-mmproj-Q8_0.gguf -ngl 99 -fa on -c 32768 --port 8080 ``` ```bash curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ "messages": [{"role": "user", "content": [ {"type": "text", "text": "What is in this image?"}, {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/4/47/PNG_transparency_demonstration_1.png"}} ]}], "chat_template_kwargs": {"enable_thinking": false} }' ``` ### Use upstream llama.cpp The `Q2_0` file also runs on stock llama.cpp from release [b11556](https://github.com/ggml-org/llama.cpp/releases/tag/b11556) onward, which added the Hadamard activation runtime these weights need. Older builds lack that runtime and give wrong output without any warning, so use a current release. Text and image input both work: ```bash hf download prism-ml/Ternary-Bonsai-2-2B-gguf Ternary-Bonsai-2-2B-Q2_0.gguf Ternary-Bonsai-2-2B-mmproj-Q8_0.gguf --local-dir . llama-cli -m Ternary-Bonsai-2-2B-Q2_0.gguf -ngl 99 -fa on \ -p "Explain in three sentences why the sky is blue." llama-mtmd-cli -m Ternary-Bonsai-2-2B-Q2_0.gguf \ --mmproj Ternary-Bonsai-2-2B-mmproj-Q8_0.gguf -ngl 99 \ --image photo.jpg -p "Describe this image." ``` `llama-server` takes the same `-m` and `--mmproj` arguments as above. ## Best Practices ### Generation Parameters We recommend these sampling settings. They are the same for every packing and for MLX: > - Thinking mode (default): `temperature=1.0`, `top_p=0.95`, `top_k=20`, `min_p=0.0`, `presence_penalty=1.0`, `repetition_penalty=1.0` > - Non-thinking mode: `temperature=0.7`, `top_p=0.80`, `top_k=20`, `min_p=0.0`, `presence_penalty=1.5`, `repetition_penalty=1.0` Apply the penalties over the last 64 tokens (llama.cpp's default window). Presence penalty has no GGUF metadata key, so pass it on the command line or per request. ```bash # thinking on ./bin/llama-server -m Ternary-Bonsai-2-2B-PQ2_0.gguf -ngl 99 -fa on \ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 --presence-penalty 1.0 --repeat-penalty 1.0 --repeat-last-n 64 # thinking off: start the server with these values, and send # "chat_template_kwargs": {"enable_thinking": false} with each request ./bin/llama-server -m Ternary-Bonsai-2-2B-PQ2_0.gguf -ngl 99 -fa on \ --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0 --presence-penalty 1.5 --repeat-penalty 1.0 --repeat-last-n 64 ``` Avoid greedy decoding (`temperature=0`) with thinking on; it can get stuck reasoning on some prompts. ### Output Length Give thinking mode room to finish. Even simple prompts can use 3,000+ thinking tokens, so allow at least 8,192 output tokens with thinking on. For short answers, turn thinking off. ### System Prompt You can use a simple system prompt such as: ``` You are a helpful assistant ``` ### MLX On Apple Silicon you can also use the MLX build of the same weights, [Ternary-Bonsai-2-2B-mlx-2bit](https://huggingface.co/prism-ml/Ternary-Bonsai-2-2B-mlx-2bit). It loads natively in mlx-vlm 0.7.8 and newer (the released mlx-lm cannot load it yet). Pass the values above explicitly: mlx-vlm's Python `generate()` is greedy by default. Its model card has the commands. ### Choosing a Packing - **PQ2_0**: the default; cheaper to unpack, and the pack measured on Apple Silicon. - **PTQ1_0**: the smallest file; use it where memory is tightest. - **Q2_0**: an alternative layout, and the one format upstream llama.cpp can also run (b11556 or newer). ## Speculative Decoding The 2B ships without a draft model. At this size a decode step is already cheap, and in our tests speculative decoding with a draft head was slower than plain decoding on everyday prompts, so plain decoding is the recommended setup. ## License Apache 2.0, following the base model.