---
title: MS-LongWriter-GLM4-9B-Chat
canonical_url: "https://www.modelscope.cn/models/swift/MS-LongWriter-GLM4-9B-Chat"
md_url: "https://www.modelscope.cn/models/swift/MS-LongWriter-GLM4-9B-Chat.md"
repository: swift/MS-LongWriter-GLM4-9B-Chat
last_updated: 2024-09-23
pipeline_tag: nli
tasks:
  - nli
model_type:
  - chatglm
architectures:
  - ChatGLMForConditionalGeneration
parameters: 9.4B
tensor_type:
  - BF16
library_name:
  - safetensors
  - pytorch
frameworks:
  - Pytorch
language:
  - en
  - zh
inference_backends:
  - "deploy_task text/emb"
  - "lmdeploy_turbomind 0.9.1"
  - "vllm 0.9.2"
downloads: 199
stars: 3
tags:
  - "Long Context"
  - chatglm
---

# MS-LongWriter-GLM4-9B-Chat

> MS-LongWriter-GLM4-9B-Chat - swift 在 ModelScope 开源的模型。MS-LongWriter-GLM4-9B-Chat

swift/MS-LongWriter-GLM4-9B-Chat 是 ModelScope 魔搭社区上的 9.4B 参数nli模型，可用 deploy_task text/emb、lmdeploy_turbomind 0.9.1、vllm 0.9.2 部署。

- **Repository**: swift/MS-LongWriter-GLM4-9B-Chat
- **Tasks**: nli
- **Parameters**: 9.4B
- **Inference backends**: deploy_task text/emb, lmdeploy_turbomind 0.9.1, vllm 0.9.2
- **Tags**: Long Context, chatglm
- **Downloads**: 199
- **Stars**: 3
- **Last updated**: 2024-09-23

Source: https://www.modelscope.cn/models/swift/MS-LongWriter-GLM4-9B-Chat

---

# MS-LongWriter-GLM4-9B-Chat

<p align="center">
  🤖 <a href="https://modelscope.cn/datasets/swift/longwriter-6k-filtered" target="_blank">[LongWriter Dataset] </a> • 💻 <a href="https://github.com/THUDM/LongWriter" target="_blank">[Github Repo]</a> • 📃 <a href="https://arxiv.org/abs/2408.07055" target="_blank">[LongWriter Paper]</a> 
</p>

MS-LongWriter-GLM4-9B-Chat is trained based on [glm4-9b-chat](https://modelscope.cn/models/ZhipuAI/glm-4-9b-chat), and is capable of generating 10,000+ words at once.

Compared to the work in the original LongWriter paper, MS-LongWriter-GLM4-9B-Chat begins training directly from the GLM4-9B-Chat model, while performing significant distillation on the [LongWriter-6k](https://modelscope.cn/datasets/ZhipuAI/LongWriter-6k) to obtain 666 high-quality samples.


## Datasets
1. [LongWriter-6k-filtered](https://modelscope.cn/datasets/swift/longwriter-6k-filtered), which is based on the [LongWriter-6k](https://modelscope.cn/datasets/ZhipuAI/LongWriter-6k)
2. [Magpie-Qwen2-Pro-200K-Chinese](https://modelscope.cn/datasets/AI-ModelScope/Magpie-Qwen2-Pro-200K-Chinese) , random sampling 6k examples.
3. [Magpie-Qwen2-Pro-200K-English](https://modelscope.cn/datasets/AI-ModelScope/Magpie-Qwen2-Pro-200K-English) , random sampling 6k examples.


## Model

We use [ms-swift](https://github.com/modelscope/swift) to fine-tune the GLM4-9B-Chat model.

1. Installation
```python
pip install ms-swift[llm]
```

2. Fine-tuning  

Envs:  
```text
Nvidia A100(80G) x 4
```

Run:  
```shell
swift sft \
    --model_type glm4-9b-chat \
    --dataset longwriter-6k-filtered#666 qwen2-pro-zh#6660 qwen2-pro-en#6660 \
    --max_length 28672 \
    --num_train_epochs 2 \
    --eval_steps 200 \
    --batch_size 1 \
    --gradient_accumulation_steps 64 \
    --gradient_checkpointing true \
    --warmup_ratio 0.1 \
    --learning_rate 1e-5 \
    --sft_type full \
    --loss_name long-ce \
    --check_dataset_strategy warning \
    --save_only_model false \
    --save_total_limit -1 \
    --lazy_tokenize true \
    --dataloader_num_workers 1 \
    --resume_only_model true \
    --neftune_noise_alpha
```


## Evaluation

Refer to [LongWriter Evaluation](https://github.com/modelscope/evalscope/tree/main/evalscope/third_party/longbench_write) from the [EvalScope](https://github.com/modelscope/evalscope).