--- title: Qwen3.5-0.8B-weight-space-programming canonical_url: "https://www.modelscope.cn/models/RikiyaT/Qwen3.5-0.8B-weight-space-programming" md_url: "https://www.modelscope.cn/models/RikiyaT/Qwen3.5-0.8B-weight-space-programming.md" repository: RikiyaT/Qwen3.5-0.8B-weight-space-programming last_updated: 2026-11-04 license: apache-2.0 model_type: - qwen3_5 architectures: - Qwen3_5ForConditionalGeneration base_model: - Qwen/Qwen3.5-0.8B base_model_relation: finetune parameters: 873.4M tensor_type: - BF16 - F32 library_name: - transformer - safetensors - pytorch frameworks: - pytorch downloads: 31 stars: 3 tags: - ml-intern - weight-space-programming --- # Qwen3.5-0.8B-weight-space-programming > Qwen3.5-0.8B-weight-space-programming - RikiyaT 在 ModelScope 开源的模型。Qwen3.5-0.8B, edited by an LLM with no training (Weight-Space Programming) RikiyaT/Qwen3.5-0.8B-weight-space-programming 是 ModelScope 魔搭社区上的 873.4M 参数机器学习模型，采用 apache-2.0 许可，基于 Qwen/Qwen3.5-0.8B 构建。 - **Repository**: RikiyaT/Qwen3.5-0.8B-weight-space-programming - **License**: apache-2.0 - **Parameters**: 873.4M - **Base model**: Qwen/Qwen3.5-0.8B - **Tags**: ml-intern, weight-space-programming - **Downloads**: 31 - **Stars**: 3 - **Last updated**: 2026-11-04 Source: https://www.modelscope.cn/models/RikiyaT/Qwen3.5-0.8B-weight-space-programming --- # Qwen3.5-0.8B, edited by an LLM with no training (Weight-Space Programming) [![Built with ML Intern](https://huggingface.co/datasets/huggingface/badges/resolve/main/built-with-ml-intern-sm.svg)](https://hf.co/chat?mode=ml-intern) This is the edited model from Experiment 8 of **["Sparks of Weight-Space Programming"](https://rikiyat.github.io/blog/weight-space-programming/)**. We asked a frontier LLM ("Astra") to make Qwen3.5-0.8B better at math by editing its weights directly, with no gradient descent and no training. Working in a Python session over the weights, with a budget of 50 forward passes on practice problems from the GSM8K and MATH training sets, Astra traced the base model's thinking loops to one direction written by the MLPs of layers 15 and 19, and subtracted it from 64 output columns of `down_proj` in each layer (131,072 weights, 0.016% of the model). The base model loops until the token limit on almost every math problem. The edited one mostly doesn't. **🎮 Try it live:** [akhaliq/weight-space-programming-demo](https://huggingface.co/spaces/akhaliq/weight-space-programming-demo) — type a problem, watch the base model loop while the edited one solves it. ## Results From the post (thinking on, up to 32,768 tokens, several samples per problem): | Qwen3.5-0.8B | MATH-500 (%) | GSM8K (%) | IFEval (%) | Hit token limit (%) | |---|---|---|---|---| | Base | 7.9 | 13.8 | 21.0 | 93.0 | | **Edited (this model)** | **48.1** | **67.1** | **62.3** | **33.1** | **Independent re-run of this checkpoint:** full MATH-500, 500 problems × 4 samples = 2,000 generations (43.4M generated tokens, 51 min on an H200), with the post's protocol: vLLM, thinking on, temperature 0.6, top_p 0.95, top_k 20, presence_penalty 1.5, max 32,768 new tokens, seed 0, last `\boxed{}` graded with math-verify 0.9.0. The edited model scored **51.0%**, with **31.6%** of answers hitting the token limit. A 25-problem smoke test of the base model gave 5.0%, with 95% hitting the limit. Raw scores, per-problem results and sample generations: [`eval/`](https://huggingface.co/RikiyaT/Qwen3.5-0.8B-weight-space-programming/tree/main/eval). ## Verifying the weights Astra's edit is published as a bit-exact diff in [`RikiyaT/lm_understand-logs`](https://github.com/RikiyaT/lm_understand-logs) (`exp8_math/runs/p50c/edit/`). This checkpoint was rebuilt with `exp8_math/code/apply_edit.py`, which **sha256-verifies every rebuilt tensor against the checkpoint evaluated in the post**. Only these two tensors differ from `Qwen/Qwen3.5-0.8B` @ `2fc06364715b967f1860aea9cf38778875588b17`: | Tensor | sha256 of the raw bfloat16 bytes | |---|---| | `model.language_model.layers.15.mlp.down_proj.weight` | `82e2d733ea64feaeb4af93a1e065641d3df98b494a8b7576ef2f52e90fe2ee53` | | `model.language_model.layers.19.mlp.down_proj.weight` | `8ef3c61bc36af113ad58e89dbb0a4a14dcba5a45cb750c15ab515c82ddd12eb2` | ## Reproduction This checkpoint and the re-run above were built by [akhaliq](https://huggingface.co/akhaliq) with [ML Intern](https://hf.co/chat?mode=ml-intern) on Hugging Face Jobs, 2026-11-02: the edit was rebuilt and hash-verified on CPU, and the benchmark run on an NVIDIA H200 with vLLM 0.33.0 / transformers 5.20.2 / math-verify 0.9.0, following `exp8_math/code/evaluate.py` (thinking on, `think35` sampling, last-`\boxed{}` grading). One substitution: the post evaluated a text-only export of the model, and this checkpoint is the full multimodal release. The language-model tensors are identical (hash-verified), and text prompts do not use the vision tower. ## Usage ```python from vllm import LLM, SamplingParams from transformers import AutoTokenizer ckpt = "RikiyaT/Qwen3.5-0.8B-weight-space-programming" tok = AutoTokenizer.from_pretrained(ckpt) llm = LLM(model=ckpt, dtype="bfloat16", max_model_len=34816) prompt = tok.apply_chat_template( [{"role": "user", "content": "What is $2^{10} + 24$?\nPlease reason step by step, and put your final answer within \\boxed{}."}], tokenize=False, add_generation_prompt=True, enable_thinking=True) sp = SamplingParams(n=1, temperature=0.6, top_p=0.95, top_k=20, presence_penalty=1.5, max_tokens=32768, seed=0) print(llm.generate([prompt], sp)[0].outputs[0].text) ``` ## Provenance - Blog post: https://rikiyat.github.io/blog/weight-space-programming/ - Logs, diffs, and rebuild code: https://github.com/RikiyaT/lm_understand-logs (`exp8_math/`, CC BY 4.0 / MIT) - Demo: [akhaliq/weight-space-programming-demo](https://huggingface.co/spaces/akhaliq/weight-space-programming-demo) - Re-run (this repo's `eval/`): MATH-500 (HuggingFaceH4/MATH-500), math-verify 0.9.0 - Base model: [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B), Apache 2.0