---
title: Qwen2.5-7B-Instruct-kowiki-qa-4bit
canonical_url: "https://www.modelscope.cn/models/mlx-community/Qwen2.5-7B-Instruct-kowiki-qa-4bit"
md_url: "https://www.modelscope.cn/models/mlx-community/Qwen2.5-7B-Instruct-kowiki-qa-4bit.md"
repository: mlx-community/Qwen2.5-7B-Instruct-kowiki-qa-4bit
last_updated: 2024-12-21
license: apache-2.0
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - qwen2
architectures:
  - Qwen2ForCausalLM
parameters: 1.2B
tensor_type:
  - F16
  - U32
library_name:
  - mlx
  - safetensors
  - pytorch
frameworks:
  - pytorch
language:
  - ko
  - en
inference_backends:
  - "deploy_task text/emb"
  - "lmdeploy 0.9.1"
  - "lmdeploy_turbomind 0.9.1"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 1002
stars: 0
tags:
  - text-generation
  - qwen2.5
  - korean
  - instruct
  - mlx
  - 4bit
---

# Qwen2.5-7B-Instruct-kowiki-qa-4bit

> Qwen2.5-7B-Instruct-kowiki-qa-4bit - mlx-community 在 ModelScope 开源的模型。Qwen2.5-7B-Instruct-kowiki-qa-4bit mlx convert model Original model is beomi/Qwen2.5-7B-Instruct-kowiki-qa

mlx-community/Qwen2.5-7B-Instruct-kowiki-qa-4bit 是 ModelScope 魔搭社区上的 1.2B 参数text-generation模型，采用 apache-2.0 许可，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。

- **Repository**: mlx-community/Qwen2.5-7B-Instruct-kowiki-qa-4bit
- **License**: apache-2.0
- **Tasks**: text-generation
- **Parameters**: 1.2B
- **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2
- **Tags**: text-generation, qwen2.5, korean, instruct, mlx, 4bit
- **Downloads**: 1002
- **Stars**: 0
- **Last updated**: 2024-12-21

Source: https://www.modelscope.cn/models/mlx-community/Qwen2.5-7B-Instruct-kowiki-qa-4bit

---

## Qwen2.5-7B-Instruct-kowiki-qa-4bit mlx convert model
- Original model is [beomi/Qwen2.5-7B-Instruct-kowiki-qa](https://huggingface.co/beomi/Qwen2.5-7B-Instruct-kowiki-qa)


## Requirement
- `pip install mlx-lm`

## Usage
- [Generate with CLI](https://github.com/ml-explore/mlx-examples/blob/main/llms/README.md#command-line)
    ```bash
    mlx_lm.generate --model mlx-community/Qwen2.5-7B-Instruct-kowiki-qa-4bit --prompt "하늘이 파란 이유가 뭐야?"
    ```

- [In Python](https://github.com/ml-explore/mlx-examples/blob/main/llms/README.md#python-api)
    ```python
    from mlx_lm import load, generate
    
    model, tokenizer = load(
        "mlx-community/Qwen2.5-7B-Instruct-kowiki-qa-4bit",
        tokenizer_config={"trust_remote_code": True},
    )

    prompt = "하늘이 파란 이유가 뭐야?"
    
    messages = [
        {"role": "system", "content": "당신은 친철한 챗봇입니다."},
        {"role": "user", "content": prompt},
    ]
    prompt = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )
    
    text = generate(
        model,
        tokenizer,
        prompt=prompt,
        # verbose=True,
        # max_tokens=8196,
        # temp=0.0,
    )
    ```

- [OpenAI Compitable HTTP Server](https://github.com/ml-explore/mlx-examples/blob/main/llms/mlx_lm/SERVER.md)
    ```bash
    mlx_lm.server --model mlx-community/Qwen2.5-7B-Instruct-kowiki-qa-4bit --host 0.0.0.0
    ```

    ```python
    import openai


    client = openai.OpenAI(
        base_url="http://localhost:8080/v1",
    )

    prompt = "하늘이 파란 이유가 뭐야?"

    messages = [
        {"role": "system", "content": "당신은 친절한 챗봇입니다.",},
        {"role": "user", "content": prompt},
    ]
    res = client.chat.completions.create(
        model='mlx-community/Qwen2.5-7B-Instruct-kowiki-qa-4bit',
        messages=messages,
        temperature=0.2,
    )
    
    print(res.choices[0].message.content)
    ```