--- title: internlm-chat-20b-4bit canonical_url: "https://www.modelscope.cn/models/Shanghai_AI_Laboratory/internlm-chat-20b-4bit" md_url: "https://www.modelscope.cn/models/Shanghai_AI_Laboratory/internlm-chat-20b-4bit.md" repository: Shanghai_AI_Laboratory/internlm-chat-20b-4bit chinese_name: "书生·浦语-对话-20B-4bit" last_updated: 2025-02-26 license: apache-2.0 pipeline_tag: text-generation tasks: - text-generation model_type: - internlm architectures: - InternLMForCausalLM library_name: - pytorch frameworks: - pytorch inference_backends: - "deploy_task text/emb" - "lmdeploy 0.9.1" - "lmdeploy_turbomind 0.9.1" - "vllm 0.9.2" downloads: 19361 stars: 6 --- # internlm-chat-20b-4bit > internlm-chat-20b-4bit - Shanghai_AI_Laboratory 在 ModelScope 开源的模型。上海人工智能实验室与商汤科技联合香港中文大学和复旦大学正式推出书生·浦语200亿参数模型版本 InternLM-20B ，InternLM-20B 在超过 2.3T Tokens 包含高质量英文、中文和代码的数据上进行预训练，其中 Chat 版本还经过了 SFT 和 RLHF 训练，使其能够更好、更安全地满足用户的需求。 Shanghai_AI_Laboratory/internlm-chat-20b-4bit 是 ModelScope 魔搭社区上的text-generation模型，采用 apache-2.0 许可，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。 - **Repository**: Shanghai_AI_Laboratory/internlm-chat-20b-4bit - **License**: apache-2.0 - **Tasks**: text-generation - **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, vllm 0.9.2 - **Downloads**: 19361 - **Stars**: 6 - **Last updated**: 2025-02-26 Source: https://www.modelscope.cn/models/Shanghai_AI_Laboratory/internlm-chat-20b-4bit ---
[LMDeploy](https://github.com/InternLM/lmdeploy) supports LLM model inference of 4-bit weight, with the minimum requirement for NVIDIA graphics cards being sm80, such as A10, A100, Geforce 30/40 series. Before proceeding with the inference of `internlm-chat-20b-4bit`, please ensure that lmdeploy is installed. ```shell pip install 'lmdeploy>=0.0.11' ``` ## Inference Please download `internlm-chat-20b-4bit` model as follows, ```shell git-lfs install git clone https://huggingface.co/internlm/internlm-chat-20b-4bit ``` As demonstrated in the command below, first convert the model's layout using `turbomind.deploy`, and then you can interact with the AI assistant in the terminal ```shell # Convert the model's layout and store it in the default path, ./workspace. python3 -m lmdeploy.serve.turbomind.deploy \ --model-name internlm-chat-20b \ --model-path ./internlm-chat-20b-4bit \ --model-format awq \ --group-size 128 # inference python3 -m lmdeploy.turbomind.chat ./workspace ``` ## Serve with gradio If you wish to interact with the model via web UI, please initiate the gradio server as indicated below: ```shell python3 -m lmdeploy.serve.gradio.app ./workspace --server_name {ip_addr} --server_port {port} ``` Subsequently, you can open the website `http://{ip_addr}:{port}` in your browser and interact with the model. Besides serving with gradio, there are two more serving methods. One is serving with Triton Inference Server (TIS), and the other is an OpenAI-like server named as `api_server`. Please refer to the [user guide](https://github.com/InternLM/lmdeploy#quick-start) for detailed information if you are interested. ## Inference Performance LMDeploy provides scripts for benchmarking `token throughput` and `request throughput`. `token throughput` tests the speed of generating new tokens, given a specified number of prompt tokens and completion tokens, while `request throughput` measures the number of requests processed per minute with real dialogue data. We conducted benchmarks on `internlm-chat-20b-4bit`. And `token_throughput` was measured by setting 256 prompt tokens and generating 512 tokens in response on A100-80G. **Note**: The `session_len` in `workspace/triton_models/weights/config.ini` is changed to `2056` in our test. | batch | tensor parallel | prompt_tokens | completion_tokens | thr_per_proc(token/s) | rpm (req/min) | mem_per_proc(GB) | |-------|-----------------|---------------|-------------------|-----------------------|---------------|------------------| | 1 | 1 | 256 | 512 | 88.77 | - | 15.65 | | 16 | 1 | 256 | 512 | 792.7 | 220.23 | 51.46 | ### token throughput Run the following command, ```shell python benchmark/profile_generation.py \ --model-path ./workspace \ --concurrency 1 8 16 --prompt-tokens 256 512 512 1024 --completion-tokens 512 512 1024 1024 --dst-csv ./token_throughput.csv ``` You will find the `token_throughput` metrics in `./token_throughput.csv` | batch | prompt_tokens | completion_tokens | thr_per_proc(token/s) | thr_per_node(token/s) | rpm(req/min) | mem_per_proc(GB) | mem_per_gpu(GB) | mem_per_node(GB) | |-------|---------------|-------------------|-----------------------|-----------------------|--------------|------------------|-----------------|------------------| | 1 | 256 | 512 | 88.77 | 710.12 | - | 15.65 | 15.65 | 125.21 | | 1 | 512 | 512 | 83.89 | 671.15 | - | 15.68 | 15.68 | 125.46 | | 1 | 512 | 1024 | 80.19 | 641.5 | - | 15.68 | 15.68 | 125.46 | | 1 | 1024 | 1024 | 72.34 | 578.74 | - | 15.75 | 15.75 | 125.96 | | 1 | 1 | 2048 | 80.69 | 645.55 | - | 15.62 | 15.62 | 124.96 | | 8 | 256 | 512 | 565.21 | 4521.67 | - | 32.37 | 32.37 | 258.96 | | 8 | 512 | 512 | 489.04 | 3912.33 | - | 32.62 | 32.62 | 260.96 | | 8 | 512 | 1024 | 467.23 | 3737.84 | - | 32.62 | 32.62 | 260.96 | | 8 | 1024 | 1024 | 383.4 | 3067.19 | - | 33.06 | 33.06 | 264.46 | | 8 | 1 | 2048 | 487.74 | 3901.93 | - | 32.12 | 32.12 | 256.96 | | 16 | 256 | 512 | 792.7 | 6341.6 | - | 51.46 | 51.46 | 411.71 | | 16 | 512 | 512 | 639.4 | 5115.17 | - | 51.93 | 51.93 | 415.46 | | 16 | 512 | 1024 | 591.39 | 4731.09 | - | 51.93 | 51.93 | 415.46 | | 16 | 1024 | 1024 | 449.11 | 3592.85 | - | 52.06 | 52.06 | 416.46 | | 16 | 1 | 2048 | 620.5 | 4964.02 | - | 51 | 51 | 407.96 | ### request throughput LMDeploy uses ShareGPT dataset to test request throughput. Try the next commands, and you will get the `rpm` (request per minute) metric. ``` # download the ShareGPT dataset wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json # python profile_throughput.py \ ShareGPT_V3_unfiltered_cleaned_split.json \ ./workspace \ --concurrency 16 ```