--- title: MS-LongWriter-Qwen2-7B-Instruct canonical_url: "https://www.modelscope.cn/models/swift/MS-LongWriter-Qwen2-7B-Instruct" md_url: "https://www.modelscope.cn/models/swift/MS-LongWriter-Qwen2-7B-Instruct.md" repository: swift/MS-LongWriter-Qwen2-7B-Instruct last_updated: 2024-09-23 pipeline_tag: text-generation tasks: - text-generation model_type: - qwen2 architectures: - Qwen2ForCausalLM parameters: 7.6B tensor_type: - BF16 library_name: - safetensors - pytorch frameworks: - Pytorch language: - en - zh inference_backends: - "deploy_task text/emb" - "lmdeploy 0.9.1" - "lmdeploy_turbomind 0.9.1" - "sglang 0.5.2" - "vllm 0.9.2" downloads: 162 stars: 0 tags: - "Long Context" - qwen2 --- # MS-LongWriter-Qwen2-7B-Instruct > MS-LongWriter-Qwen2-7B-Instruct - swift 在 ModelScope 开源的模型。MS-LongWriter-Qwen2-7B-Instruct swift/MS-LongWriter-Qwen2-7B-Instruct 是 ModelScope 魔搭社区上的 7.6B 参数text-generation模型，可用 deploy_task text/emb、lmdeploy 0.9.1、lmdeploy_turbomind 0.9.1 部署。 - **Repository**: swift/MS-LongWriter-Qwen2-7B-Instruct - **Tasks**: text-generation - **Parameters**: 7.6B - **Inference backends**: deploy_task text/emb, lmdeploy 0.9.1, lmdeploy_turbomind 0.9.1, sglang 0.5.2, vllm 0.9.2 - **Tags**: Long Context, qwen2 - **Downloads**: 162 - **Stars**: 0 - **Last updated**: 2024-09-23 Source: https://www.modelscope.cn/models/swift/MS-LongWriter-Qwen2-7B-Instruct --- # MS-LongWriter-Qwen2-7B-Instruct

🤖 [LongWriter Dataset] • 💻 [Github Repo] • 📃 [LongWriter Paper]

MS-LongWriter-Qwen2-7B-Instruct is trained based on [https://modelscope.cn/models/qwen/Qwen2-7B-Instruct](https://modelscope.cn/models/qwen/Qwen2-7B-Instruct), and is capable of generating 10,000+ words at once. MS-LongWriter-Qwen2-7B-Instruct begins training directly from the Qwen2-7B-Instruct, while performing significant distillation on the [LongWriter-6k](https://modelscope.cn/datasets/ZhipuAI/LongWriter-6k) to obtain 666 high-quality samples. ## Datasets 1. [LongWriter-6k-filtered](https://modelscope.cn/datasets/swift/longwriter-6k-filtered), which is based on the [LongWriter-6k](https://modelscope.cn/datasets/ZhipuAI/LongWriter-6k) 2. [Magpie-Qwen2-Pro-200K-Chinese](https://modelscope.cn/datasets/AI-ModelScope/Magpie-Qwen2-Pro-200K-Chinese) , random sampling 6k examples. 3. [Magpie-Qwen2-Pro-200K-English](https://modelscope.cn/datasets/AI-ModelScope/Magpie-Qwen2-Pro-200K-English) , random sampling 6k examples. ## Model We use [ms-swift](https://github.com/modelscope/swift) to fine-tune the Qwen2-7B-Instruct model. 1. Installation ```python pip install ms-swift[llm] ``` 2. Fine-tuning Envs: ```text Nvidia A100(80G) x 4 ``` Run: ```shell swift sft \ --model_type qwen2-7b-instruct \ --dataset longwriter-6k-filtered#666 qwen2-pro-zh#6660 qwen2-pro-en#6660 \ --max_length 28672 \ --num_train_epochs 2 \ --eval_steps 200 \ --batch_size 1 \ --gradient_accumulation_steps 64 \ --gradient_checkpointing true \ --warmup_ratio 0.1 \ --learning_rate 1e-5 \ --sft_type full \ --loss_name long-ce \ --check_dataset_strategy warning \ --save_only_model false \ --save_total_limit -1 \ --lazy_tokenize true \ --dataloader_num_workers 1 \ --resume_only_model true \ --neftune_noise_alpha 5 \ --use_flash_attn true ``` ## Evaluation Refer to [LongWriter Evaluation](https://github.com/modelscope/evalscope/tree/main/evalscope/third_party/longbench_write) from the [EvalScope](https://github.com/modelscope/evalscope).