--- title: DeepSeek-V3.1-Terminus-gguf-q2ks-mixed-AutoRound canonical_url: "https://www.modelscope.cn/models/Intel/DeepSeek-V3.1-Terminus-gguf-q2ks-mixed-AutoRound" md_url: "https://www.modelscope.cn/models/Intel/DeepSeek-V3.1-Terminus-gguf-q2ks-mixed-AutoRound.md" repository: Intel/DeepSeek-V3.1-Terminus-gguf-q2ks-mixed-AutoRound last_updated: 2025-09-25 base_model: - deepseek-ai/DeepSeek-V3.1-Terminus base_model_relation: quantized library_name: - pytorch - gguf frameworks: - pytorch downloads: 10225 stars: 0 tags: - gguf --- # DeepSeek-V3.1-Terminus-gguf-q2ks-mixed-AutoRound > DeepSeek-V3.1-Terminus-gguf-q2ks-mixed-AutoRound - Intel 在 ModelScope 开源的模型。This model is a mixed gguf q2ks format of deepseek-ai/DeepSeek-V3.1-Terminus generated by intel/auto-round algorithm. Embedding layer and lm-head layer are fallback to 8 bits and non… Intel/DeepSeek-V3.1-Terminus-gguf-q2ks-mixed-AutoRound 是 ModelScope 魔搭社区上的机器学习模型，基于 deepseek-ai/DeepSeek-V3.1-Terminus 构建。 - **Repository**: Intel/DeepSeek-V3.1-Terminus-gguf-q2ks-mixed-AutoRound - **Base model**: deepseek-ai/DeepSeek-V3.1-Terminus - **Tags**: gguf - **Downloads**: 10225 - **Stars**: 0 - **Last updated**: 2025-09-25 Source: https://www.modelscope.cn/models/Intel/DeepSeek-V3.1-Terminus-gguf-q2ks-mixed-AutoRound --- ## Model Details This model is a mixed gguf q2ks format of [deepseek-ai/DeepSeek-V3.1-Terminus](https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus) generated by [intel/auto-round](https://github.com/intel/auto-round) algorithm. Embedding layer and lm-head layer are fallback to 8 bits and non expert layers are fallback to 4 bits. Please refer to Section `Generate the model` for more details. Please follow the license of the original model. ## How To Use Llama.cpp inference ~~~bash ./llama-cli -hf Intel/DeepSeek-V3.1-Terminus-gguf-q2ks-mixed-AutoRound ~~~ ~~~bash > Write a quick sort algorithm. Here's a quick sort algorithm implemented in Python: ```python def quicksort(arr): """ Sorts an array using the quicksort algorithm. Args: arr: List of comparable elements Returns: Sorted list """ if len(arr) <= 1: return arr pivot = arr[len(arr) // 2] # Choose middle element as pivot left = [x for x in arr if x < pivot] middle = [x for x in arr if x == pivot] right = [x for x in arr if x > pivot] return quicksort(left) + middle + quicksort(right) # Example usage: if __name__ == "__main__": # Test the algorithm test_array = [64, 34, 25, 12, 22, 11, 90] print("Original array:", test_array) sorted_array = quicksort(test_array) print("Sorted array:", sorted_array) ``` Here's an in-place version that uses less memory: ```python def quicksort_inplace(arr, low=0, high=None): """ In-place quicksort implementation. Args: arr: List to be sorted low: Starting index high: Ending index """ if high is None: high = len(arr) - 1 if low < high: # Partition the array and get pivot index pivot_index = partition(arr, low, high) # Recursively sort elements before and after partition quicksort_inplace(arr, low, pivot_index - 1) quicksort_inplace(arr, pivot_index + 1, high) def partition(arr, low, high): """ Partition the array using the last element as pivot. Args: arr: List to partition low: Starting index high: Ending index Returns: Index of the pivot element after partitioning """ pivot = arr[high] i = low - 1 # Index of smaller element for j in range(low, high): if arr[j] <= pivot: i += 1 arr[i], arr[j] = arr[j], arr[i] arr[i + 1], arr[high] = arr[high], arr[i + 1] return i + 1 # Example usage: if __name__ == "__main__": test_array = [64, 34, 25, 12, 22, 11, 90] print("Original array:", test_array) quicksort_inplace(test_array) print("Sorted array:", test_array) ``` ## Key Features: 1. **First version (easy to understand):** - Uses list comprehensions to create left, middle, and right partitions - More readable but uses extra memory 2. **Second version (in-place):** - More memory efficient - Uses the Lomuto partition scheme - Modifies the original array ## Time Complexity: - **Best/Average case:** O(n log n) - **Worst case:** O(n²) - when pivot is always the smallest or largest element ## Space Complexity: - **First version:** O(n) - due to creating new lists - **Second version:** O(log n) - due to recursion stack The algorithm works by selecting a 'pivot' element and partitioning the array around it, then recursively sorting the sub-arrays. ~~~ ### Generate the model Here is the sample command to reproduce the model ```python import torch from auto_round import AutoRound from auto_round.utils import llm_load_model model_name = "deepseek-ai/DeepSeek-V3.1-Terminus" model, tokenizer,_=llm_load_model(model_name,trust_remote_code=False,device="cpu") layer_config = {} for n, m in model.named_modules(): if isinstance(m,torch.nn.Embedding)： layer_config[n] = {"bits": 8} if isinstance(m, torch.nn.Linear): if n=="lm_head": layer_config[n] = {"bits": 8} continue if "expert" in n and "shared_experts" not in n: layer_config[n] = {"bits": 2} elif n != "lm_head": layer_config[n] = {"bits": 4} print(n, 4) ar = AutoRound(model, tokenizer=tokenizer, iters=0, scheme="gguf:q2_k_s", low_gpu_mem_usage=True, layer_config=layer_config) ar.quantize_and_save(format="gguf:q2_k_s", output_dir="tmp_autoround") ``` ## Ethical Considerations and Limitations The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs. Therefore, before deploying any applications of the model, developers should perform safety testing. ## Caveats and Recommendations Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Here are a couple of useful links to learn more about Intel's AI software: - Intel Neural Compressor [link](https://github.com/intel/neural-compressor) ## Disclaimer The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes. ## Cite @article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} } [arxiv](https://arxiv.org/abs/2309.05516) [github](https://github.com/intel/auto-round)