Blog | Hugging Face Bio | Code
# Carbon-X GGUF GGUF quantizations of [Carbon-X](https://huggingface.co/HuggingFaceBio/Carbon-X), an open 14B language model for DNA understanding and generation from the **Carbon** family, for local inference with [llama.cpp](https://github.com/ggml-org/llama.cpp). Carbon-X reads raw DNA inside an ordinary chat conversation and answers in English: it predicts whether mutations are benign or pathogenic, identifies regions that control gene activity, classifies sequences by organism group, predicts the next bases of a sequence and explains its reasoning. A hybrid tokenizer keeps a BPE vocabulary for text and encodes DNA as non-overlapping 6-mers. See the [Carbon-X model card](https://huggingface.co/HuggingFaceBio/Carbon-X) for details on the model and its limitations. > [!IMPORTANT] > These files need llama.cpp [b11742](https://github.com/ggml-org/llama.cpp/releases/tag/b11742) or newer, the first > release with support for the Carbon `HybridDNATokenizer` ([#30825](https://github.com/ggml-org/llama.cpp/pull/30825)). > Older builds load them but tokenize DNA wrongly. ## Table of Contents 1. [Available files](#available-files) 2. [How to use](#how-to-use) 3. [Evaluation](#evaluation) 4. [License](#license) 5. [Acknowledgements](#acknowledgements) ## Available files | Quant | File | Size | | --- | --- | ---: | | BF16 | `carbon-x-bf16.gguf` | 29.7 GB | | Q8_0 (recommended) | `carbon-x-q8_0.gguf` | 15.8 GB | | Q6_K | `carbon-x-q6_k.gguf` | 12.6 GB | | Q5_K_M | `carbon-x-q5_k_m.gguf` | 11.1 GB | | Q4_K_M | `carbon-x-q4_k_m.gguf` | 9.6 GB | **Q8_0 is the recommended file**: half the size of BF16, with the same benchmark scores. Use Q4_K_M when memory is tight. In every K-quant the token embedding and output tensors, which hold the DNA tokens, stay at Q8_0. ## How to use ### Install llama.cpp Any llama.cpp release from [b11742](https://github.com/ggml-org/llama.cpp/releases/tag/b11742) on works: a prebuilt binary from the [releases page](https://github.com/ggml-org/llama.cpp/releases), a package manager once it ships b11742 or newer (`brew install llama.cpp`, `winget install llama.cpp`, ...), or a build from source: ```bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp cmake -B build -DGGML_CUDA=ON # drop -DGGML_CUDA=ON for a CPU-only build, or use -DGGML_METAL=ON etc. cmake --build build -j --target llama-server llama-cli ``` The commands below assume `llama-server` and `llama-cli` are on your `PATH` (for a source build they are in `build/bin`). With `-hf`, llama.cpp downloads and caches the quant on first use. While this repo is private, log in first with `hf auth login` or set `HF_TOKEN`. To use a local file instead, download it and replace `-hf HuggingFaceBio/Carbon-X-GGUF:Q8_0` with `-m carbon-x-q8_0.gguf`: ```bash hf download HuggingFaceBio/Carbon-X-GGUF carbon-x-q8_0.gguf --local-dir . ``` ### Serving with llama-server `llama-server` provides an OpenAI-compatible API: ```bash llama-server -hf HuggingFaceBio/Carbon-X-GGUF:Q8_0 \ -ngl 99 -c 40960 --jinja \ --chat-template-kwargs '{"enable_thinking": false}' \ --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 \ --port 8080 ``` - `--jinja` uses the chat template stored in the GGUF. - Carbon-X was post-trained and evaluated with thinking disabled, and we recommend keeping `enable_thinking: false`. - `--temp 0.6 --top-p 0.95 --top-k 20 --min-p 0` are Carbon-X's default sampling settings, used for every reported result. llama.cpp's own defaults differ (temperature 0.8, top-k 40, min-p 0.05). - `-c 40960` is the model's full context. `llama-server` shares it between 4 parallel request slots by default; set `-np N` to change the number of slots and raise `-c` if each request needs the full context. - `-ngl 99` offloads every layer to the GPU; Q8_0 needs about 16 GB plus about 0.16 MB of KV cache per token. Query it with any OpenAI client: ```bash curl -s http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ "messages": [{"role": "user", "content": "Is this sequence more likely coding or non-coding? ATGGCGCTGTGGATGCGCCTCCTGCCCCTGCTGGCGCTGCTGGCCCTCTGGGGACCTGACCCAGCCGCAGCC"}] }' | jq -r '.choices[0].message.content' ``` ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8080/v1", api_key="none") response = client.chat.completions.create( model="carbon-x", # llama-server serves a single model; the name is not checked messages=[{"role": "user", "content": "Is this sequence more likely coding or non-coding? ATGGCGCTGTGGATGCGCCTCCTGCCCCTGCTGGCGCTGCTGGCCCTCTGGGGACCTGACCCAGCCGCAGCC"}], ) print(response.choices[0].message.content) ``` Example answer (Q8_0): > This sequence is more consistent with a **coding region** than with non-coding DNA. > > It begins with **ATG**, a standard start codon, and can be read in triplets from that point without encountering a stop > codon within the supplied sequence. That supports a protein-coding interpretation, though it does not by itself prove > that the sequence is translated or establish its biological function. ### Chatting in the terminal ```bash llama-cli -hf HuggingFaceBio/Carbon-X-GGUF:Q8_0 -ngl 99 -c 40960 --jinja \ --chat-template-kwargs '{"enable_thinking": false}' \ --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0 ``` Add `-st -p "your question"` for a single answer instead of an interactive chat. ### Working with DNA inputs The tokenizer switches between BPE for text and 6-mers for DNA. Paste sequences directly into a message: - **DNA is tagged automatically.** Runs of at least 6 A/C/G/T bases (either case) are wrapped in `