🤗 Hugging Face | 🤖 ModelScope | 💻 GitHub | 💬 WeChat (微信) | 🎮 Discord
> ⚡ **QuantFunc — the fastest diffusion inference engine.** Native C++/CUDA, zero Python model dependencies, **2x–11x** speedup across RTX 20/30/40/50. The official **ComfyUI-QuantFunc** plugin and engine: runtime & offline 4-bit quantization for **Z-Image, Qwen-Image, Qwen-Image-Edit and FLUX.2 Klein** pipelines. **Powered by the [QuantFunc ComfyUI plugin](https://github.com/RealJonathanYip/ComfyUI-QuantFunc) — the fastest diffusion inference engine:** - 🚀 **2x–11x speedup** over standard BF16/FP16 Python pipelines (pre-exported → even faster loading). - ⚙️ **Native C++/CUDA** (`libquantfunc.so` / `quantfunc.dll`) with **zero Python model dependencies**. - 🧩 **Dual engine** (SVDQ offline + Lighting runtime 4-bit), **zero-cost LoRA stacking**, reference-image editing & inpainting. - 🟢 **Full GPU coverage** — RTX 20/30/40/50 · A100/H100/H200/B100/B200/GB300 · RTX 6000 Ada / PRO Blackwell (CUDA 12 & 13); native **FP4** on Blackwell. 👉 **Install the plugin:** **https://github.com/RealJonathanYip/ComfyUI-QuantFunc**
The Fastest Diffusion Model Inference Engine
Up to 11x faster · Zero dependencies · LoRA at zero cost · ~130 MB binary
--- **QuantFunc** is a high-performance native inference engine for text-to-image and image editing diffusion models. Built entirely in C++/CUDA with zero Python dependency, it delivers **2x–11x speedup** over mainstream Python frameworks across NVIDIA RTX 20/30/40/50 series GPUs — no manual tuning required, no environment setup, just download and run. | Model | Hardware | Python (before) | QuantFunc | Speedup | |-------|----------|-----------------|-----------|---------| | QwenImage | RTX 4090 | 10.5 s | **0.9 s** | **11.7x** | | Z-Image-Turbo | RTX 4090 | 5.0 s | **1.2 s** | **4.2x** | > One binary. One command. The fastest image generation at your fingertips. --- ## 1. Why QuantFunc? ### 1.1 Extreme Speed — Automatically Optimized QuantFunc doesn't just run models faster — it **intelligently adapts** to your hardware. The auto-optimization engine profiles your GPU at startup, determines the optimal VRAM strategy, and configures everything transparently. **Key capabilities:** - **Full-pipeline kernel fusion** — cross-layer optimization, not just individual operators - **Intelligent VRAM management** — automatic compression and adaptive offloading - **Four-generation GPU support** — Turing (RTX 20), Ampere (RTX 30), Ada Lovelace (RTX 40), Blackwell (RTX 50) - **Multi-GPU ready** — out of the box, no configuration needed ### 1.2 Zero Setup — Download and Run No Python, no virtual environments, no `pip install`, no CUDA/PyTorch version matrix. A single self-contained binary — download, point at a model, and generate. ```bash ./quantfunc --model-dir /path/to/model --prompt "A cute cat" ``` - **~130 MB** on Linux, **~390 MB** on Windows — smaller than a typical `torch` install - **Pre-built binaries** — no compilation required - **C shared library** for integration into any language (see [Section 4.2](#42-c-api--integrate-with-any-language)) ### 1.3 Dual Engine Architecture QuantFunc offers two complementary quantization engines: #### 1.3.1 Lighting Engine — Maximum Speed An innovative **mixed-precision quantization framework** drawing on HQQ, HIGGS, and Rotation quantization. Core features: - Quantize any FP16/FP32 model on-the-fly — no calibration dataset needed - **Per-layer precision control** — fine-tune the speed–quality balance with surgical precision - **20–30% faster** inference than SVDQ - Export pre-quantized models for instant loading thereafter **Recommended workflow:** export once → run at full speed forever. #### 1.3.2 SVDQ Engine — High Quality, Plug-and-Play Load pre-quantized weight files directly with deep kernel fusion and algorithmic optimization. Core features: - **Rotation quantization** for reduced quantization error - **Runtime LoRA** hot-loading for dynamic style switching - Pre-quantized weights available from ModelScope / HuggingFace #### 1.3.3 Quick Comparison ```bash # Lighting: build a pre-quantized model from FP16 weights ./quantfunc --model-dir /path/to/base-model \ --model-backend lighting --precision-config precision.json \ --export-path /path/to/output --auto-optimize # SVDQ: download pre-quantized weights and run directly ./quantfunc --model-dir /path/to/base-model \ --transformer /path/to/svdq-weights.safetensors \ --prompt "A cute cat" --auto-optimize ``` #### 1.3.4 Precision Options Precision configs support **ratio shorthand** — e.g., `{"4": 8, "i8": 2}` means 80% 4-bit + 20% INT8, with higher-precision layers automatically placed deeper for quality. Available precisions: `"4"` (auto 4-bit), `"i4"` (INT4), `"f4"` (FP4, SM120+), `"8"` (auto 8-bit), `"i8"` (INT8), `"f8"` (FP8), `"af8wf4"` (FP8 act + FP4 wgt), `"fp16"`, `"bf16"`. See [Lighting Inference](docs/lighting-inference.md) for full config options. ### 1.4 Zero-Cost LoRA #### 1.4.1 Lighting: Build Your All-in-One Model LoRAs are merged into base weights before quantization and export. The result: - **Zero runtime overhead** — speed identical whether 1 or 100 LoRAs are merged - **Zero additional size** — no extra bytes on disk - Stack style, quality, character, and concept LoRAs into one personal AOI model #### 1.4.2 SVDQ: Smart Runtime LoRA Runtime LoRA enhanced with **Rotation quantization** and **intelligent rank compression**: > `LoRA_rank128 + LoRA_rank128 ≈ merged_rank192` (not 256) Multi-LoRA stacking stays fast and memory-efficient without sacrificing quality. --- ## 2. Supported Models | Pipeline | Use Case | Architecture | |----------|----------|--------------| | **Z-Image-Turbo** | High-speed text-to-image | Qwen3 text encoder + 30-layer transformer + VAE | | **QwenImage** | High-quality text-to-image | Qwen3 text encoder + 60-layer transformer + 3D VAE | | **QwenImage Edit** | Reference-based image editing | Vision encoder + VAE encoder/decoder + text encoder + transformer | Pipeline type is **auto-detected** from the model directory. ### 2.1 Model Download Pre-quantized models are available on [ModelScope](https://www.modelscope.cn/models/QuantFunc) and [HuggingFace](https://huggingface.co/QuantFunc). ### 2.2 Base Model & Transformer Weight Separation QuantFunc uses a **split design** — base model and transformer weights are distributed separately: - **Base model** (`*-base-model`) — text encoder, VAE, vision encoder, tokenizer, etc. - **Transformer weights** (`*.safetensors`) — quantized transformer parameters only **Why?** A single base model is compatible with all transformer weight variants in the same series. Download once, switch between different engines and configs freely. ``` QuantFunc/Qwen-Image-Edit-Series/ ├── qwen-image-edit-series-50x-below-base-model/ # Base model (RTX 30/40) │ ├── model_index.json │ ├── text_encoder/ │ ├── vae/ │ └── ... ├── qwen-image-edit-series-50x-above-base-model/ # Base model (RTX 50) │ └── ... └── transformer/ # Transformer weights ├── qwen-image-2511-50x-below-lighting-4steps.safetensors ├── qwen-image-2511-50x-below-svdq-4steps.safetensors └── ... ``` ### 2.3 GPU Variant Guide | Variant | Target GPU | Notes | |---------|------------|-------| | `50x-below` | RTX 30/40 series | Broadly compatible, recommended for most users | | `50x-above` | RTX 50 series | Blackwell-optimized, leverages FP4 tensor cores | > **Important:** Base model and transformer weights must use the **same GPU variant**. ### 2.4 Download & Use Example ```bash # 1. Download models (ModelScope) pip install modelscope python -c "from modelscope import snapshot_download; snapshot_download('QuantFunc/Qwen-Image-Edit-Series')" # 2. SVDQ inference ./quantfunc \ --model-dir /path/to/qwen-image-edit-series-50x-below-base-model \ --transformer /path/to/transformer/qwen-image-2511-50x-below-svdq-4steps.safetensors \ --ref-image input.png --prompt "Change the sky to sunset" \ --steps 4 --auto-optimize # 3. Lighting inference — same base model, just swap transformer weights ./quantfunc \ --model-dir /path/to/qwen-image-edit-series-50x-below-base-model \ --transformer /path/to/transformer/qwen-image-2511-50x-below-lighting-4steps.safetensors \ --ref-image input.png --prompt "Change the sky to sunset" \ --steps 4 --auto-optimize ``` > **Tip:** Once you have the base model, try any other transformer variant without re-downloading. --- ## 3. Quick Start Download the pre-built binary for your platform and start generating immediately. ### 3.1 Linux ```bash ./quantfunc \ --model-dir /path/to/model \ --transformer /path/to/quantized-weights.safetensors \ --prompt "A serene mountain landscape at sunset" \ --device 0 --auto-optimize ``` ### 3.2 Windows ```bat quantfunc.exe --model-dir C:\models\my-model --prompt "A cute cat" --auto-optimize ``` ### 3.3 Usage Examples #### 3.3.1 SVDQ Text-to-Image ```bash ./quantfunc --model-dir /path/to/model \ --transformer /path/to/svdq-weights.safetensors \ --prompt "A serene mountain landscape at sunset" \ --device 0 --auto-optimize --steps 4 ``` #### 3.3.2 Lighting Text-to-Image (On-the-Fly Quantization) ```bash ./quantfunc --model-dir /path/to/model \ --model-backend lighting \ --precision-config '{"transformer_blocks.attn": "i4", "transformer_blocks.img_mlp": "i4", "transformer_blocks.img_mod": "i8"}' \ --lora /path/to/lightning-lora.safetensors \ --prompt "A cute cat" --steps 4 --auto-optimize \ --quant-method higgs+hqq --rotation-block-size 256 --fused-mod ``` #### 3.3.3 Export Lighting Model ```bash ./quantfunc --model-dir /path/to/model \ --model-backend lighting --precision-config precision.json \ --lora /path/to/lora.safetensors \ --export-path /path/to/exported \ --quant-method higgs+hqq --rotation-block-size 256 --fused-mod --auto-optimize ``` #### 3.3.4 Load Exported Model ```bash ./quantfunc --model-dir /path/to/model \ --transformer /path/to/exported/transformer/model.safetensors \ --model-backend lighting --prompt "A cute cat" --steps 4 --auto-optimize --fused-mod ``` #### 3.3.5 Image Editing (QwenImage-Edit) ```bash ./quantfunc --model-dir /path/to/edit-model \ --transformer /path/to/weights.safetensors \ --ref-image input.png --prompt "Change the sky to sunset" \ --steps 4 --auto-optimize ``` --- ## 4. Integration ### 4.1 ComfyUI QuantFunc provides first-class **ComfyUI** support with full feature parity: - **Native acceleration** via `libquantfunc.so` / `quantfunc.dll` — zero PyTorch overhead - **Full features** — SVDQ, Lighting, LoRA, multi-reference editing, auto-optimize - **Process isolation** — dedicated subprocess, keeps ComfyUI stable - **Real-time progress** — step-by-step callbacks in the UI - **Multi-GPU** — assign different nodes to different GPUs > **Coming soon**: Official ComfyUI custom node package. ### 4.2 C API — Integrate with Any Language Clean C shared library with opaque handles — call QuantFunc from Python, Go, Rust, C#, Java, Node.js, Swift, and more. ```c #include "quantfunc.h" quantfunc_pipeline_t pipeline; quantfunc_create(&init_params, &pipeline); quantfunc_image_t image; quantfunc_text_to_image(pipeline, &t2i_params, &image); quantfunc_image_save(image, "output.png"); quantfunc_destroy(pipeline); ``` > **Coming soon**: Official server-side SDKs and integration examples. --- ## 5. Documentation | Guide | Description | |-------|-------------| | [Lighting Inference](docs/lighting-inference.md) | Mixed-precision inference with per-layer control | | [Lighting Export](docs/lighting-export.md) | Build zero-cost AOI models from FP16 weights | | [SVDQ Inference](docs/svdq-inference.md) | Run pre-quantized models with LoRA support | | [SVDQ Export](docs/svdq-export.md) | Export SVDQ models with merged LoRAs | --- ## 6. Features at a Glance | Feature | Description | |---------|-------------| | **Auto-Optimize** | One-click hardware profiling and optimal configuration | | **Adaptive Offload** | Run large models seamlessly on limited VRAM | | **Tiled VAE** | High-resolution decoding within memory budget | | **Smart Attention** | Auto-selects best kernel: SageAttention, FlashAttention, or cuDNN | | **Model Export** | Export optimized models for instant production loading | | **Zero-Cost LoRA** | Merge unlimited LoRAs — no speed loss, no size increase | | **ComfyUI** | Native custom nodes with full feature parity | | **Cross-Platform** | Linux + Windows, RTX 20xx / 30xx / 40xx / 50xx | --- ## 7. System Requirements | Requirement | Minimum | |-------------|---------| | **GPU** | NVIDIA RTX 20 series or newer (CC 7.5+) | | **VRAM** | 8 GB | | **RAM** | 24 GB | | **CUDA Runtime** | 13.0+ | | **cuDNN** | 9.x | | **Driver** | NVIDIA ≥ 560 | | **OS** | Linux (glibc 2.31+) or Windows 10/11 | ### 7.1 Installation (Linux) QuantFunc is self-contained — no Python, PyTorch, or OpenCV needed. You only need NVIDIA CUDA runtime and cuDNN. #### 7.1.1 Install NVIDIA Driver ```bash nvidia-smi # Requires driver ≥ 560 ``` #### 7.1.2 Install CUDA 13 Runtime ```bash # Add NVIDIA CUDA repository (Ubuntu 22.04 example) wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt update # Install runtime libraries only sudo apt install cuda-libraries-13-0 ``` If `cuda-libraries-13-0` is unavailable, install individual packages: ```bash sudo apt install libcublas-13-0 libcurand-13-0 libcusolver-13-0 libcusparse-13-0 libnvjitlink-13-0 ``` #### 7.1.3 Install cuDNN 9 ```bash sudo apt install libcudnn9-cuda-13 ``` #### 7.1.4 Verify Installation ```bash python3 -c "import ctypes; ctypes.CDLL('libquantfunc.so')" # No error = ready to go ``` ### 7.2 Installation (Windows) Download `quantfunc.exe` and place it anywhere. Runtime dependencies: - **NVIDIA Driver** ≥ 560 - **Visual C++ Redistributable** 2015-2022 ([download](https://aka.ms/vs/17/release/vc_redist.x64.exe)) - **cuDNN 9.x** ([download](https://developer.nvidia.com/cudnn)) ### 7.3 First Run — JIT Notice QuantFunc ships with **pre-compiled SASS** for RTX 30/40/50 (SM86/89/120) — instant startup. Other GPUs need a one-time PTX JIT compilation: | GPU | First Run | Subsequent Runs | |-----|-----------|-----------------| | **RTX 30/40/50** | Instant (SASS) | Instant | | **A100/H100/other** | 5–15 min (JIT) | Instant (cached) | JIT results are cached at `~/.nv/ComputeCache/` and persist across reboots. --- ## 8. License ### 8.1 Engine License QuantFunc engine (binaries and shared libraries) is proprietary software. - **Free to use** during the public beta period, with no restrictions on commercial or personal use - Redistribution, reverse engineering, and decompilation are **prohibited** - Pricing and licensing terms will be announced before the beta period ends - The author reserves the right to modify these terms at any time ### 8.2 Model Weights License Pre-quantized model weights distributed by QuantFunc are derivative works of their respective upstream models. These weights **inherit the license of the original model** (e.g., Apache 2.0 for Qwen series). Users must independently comply with the original model's license terms. QuantFunc does not grant any rights beyond those provided by the upstream model license. ---Stop waiting. Start generating.
## Community Join our community for support, updates, and discussions: - 🎮 [Discord server](https://discord.gg/jCp9TpFWcn) - 💬 Scan the QR code below to join our WeChat group: