---
title: OLMoE-1B-7B-0924-SFT
canonical_url: "https://www.modelscope.cn/models/LLM-Research/OLMoE-1B-7B-0924-SFT"
md_url: "https://www.modelscope.cn/models/LLM-Research/OLMoE-1B-7B-0924-SFT.md"
repository: LLM-Research/OLMoE-1B-7B-0924-SFT
last_updated: 2024-11-21
license: apache-2.0
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - olmoe
architectures:
  - OlmoeForCausalLM
base_model:
  - allenai/OLMoE-1B-7B-0924
base_model_relation: finetune
parameters: 6.9B
tensor_type:
  - BF16
library_name:
  - safetensors
  - pytorch
frameworks:
  - pytorch
language:
  - en
inference_backends:
  - "deploy_task text/emb"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 2204
stars: 0
tags:
  - moe
  - olmo
  - olmoe
---

# OLMoE-1B-7B-0924-SFT

> OLMoE-1B-7B-0924-SFT - LLM-Research 在 ModelScope 开源的模型。This model is an intermediate training checkpoint during post-training, after the Supervised Fine-Tuning (SFT) step. For best performance, we recommend you use the OLMoE-Instruct version.

LLM-Research/OLMoE-1B-7B-0924-SFT 是 ModelScope 魔搭社区上的 6.9B 参数text-generation模型，采用 apache-2.0 许可，基于 allenai/OLMoE-1B-7B-0924 构建，可用 deploy_task text/emb、sglang 0.5.2、vllm 0.9.2 部署。

- **Repository**: LLM-Research/OLMoE-1B-7B-0924-SFT
- **License**: apache-2.0
- **Tasks**: text-generation
- **Parameters**: 6.9B
- **Base model**: allenai/OLMoE-1B-7B-0924
- **Inference backends**: deploy_task text/emb, sglang 0.5.2, vllm 0.9.2
- **Tags**: moe, olmo, olmoe
- **Downloads**: 2204
- **Stars**: 0
- **Last updated**: 2024-11-21

Source: https://www.modelscope.cn/models/LLM-Research/OLMoE-1B-7B-0924-SFT

---

<img alt="OLMoE Logo." src="olmoe-logo.png" width="250px">

# Model Summary

This model is an intermediate training checkpoint during post-training, after the Supervised Fine-Tuning (SFT) step. For best performance, we recommend you use the [OLMoE-Instruct](https://huggingface.co/allenai/OLMoE-1B-7B-0924-Instruct) version.

- **Paper**: https://arxiv.org/abs/2409.02060
- **Pretraining** [Checkpoints](https://hf.co/allenai/OLMoE-1B-7B-0924), [Code](https://github.com/allenai/OLMo/tree/Muennighoff/MoE), [Data](https://huggingface.co/datasets/allenai/OLMoE-mix-0924) and [Logs](https://wandb.ai/ai2-llm/olmoe/reports/OLMoE-1B-7B-0924--Vmlldzo4OTcyMjU3).
- **SFT (Supervised Fine-Tuning)** [Checkpoints](https://huggingface.co/allenai/OLMoE-1B-7B-0924-SFT), [Code](https://github.com/allenai/open-instruct/tree/olmoe-sft), [Data](https://hf.co/datasets/allenai/tulu-v3.1-mix-preview-4096-OLMoE) and [Logs](https://github.com/allenai/OLMoE/blob/main/logs/olmoe-sft-logs.txt).
- **DPO/KTO (Direct Preference Optimization/Kahneman-Tversky Optimization)**, [Checkpoints](https://huggingface.co/allenai/OLMoE-1B-7B-0924-Instruct), [Preference Data](https://hf.co/datasets/allenai/ultrafeedback_binarized_cleaned), [DPO code](https://github.com/allenai/open-instruct/tree/olmoe-sft), [KTO code](https://github.com/Muennighoff/kto/blob/master/kto.py) and [Logs](https://github.com/allenai/OLMoE/blob/main/logs/olmoe-dpo-logs.txt).

Branches:
- `main`: Instruction tuned / supervised finetuned (SFT) model of https://hf.co/allenai/OLMoE-1B-7B-0924 (`main` branch)
- `load-balancing`: Ablation with load balancing loss during SFT
- `non-annealed`: Ablation starting from the checkpoint prior to annealing (branch `step1200000-tokens5033B` of https://hf.co/allenai/OLMoE-1B-7B-0924) rather than the annealed checkpoint (branch `main` of https://hf.co/allenai/OLMoE-1B-7B-0924)

# Citation

```bibtex
@misc{muennighoff2024olmoeopenmixtureofexpertslanguage,
      title={OLMoE: Open Mixture-of-Experts Language Models}, 
      author={Niklas Muennighoff and Luca Soldaini and Dirk Groeneveld and Kyle Lo and Jacob Morrison and Sewon Min and Weijia Shi and Pete Walsh and Oyvind Tafjord and Nathan Lambert and Yuling Gu and Shane Arora and Akshita Bhagia and Dustin Schwenk and David Wadden and Alexander Wettig and Binyuan Hui and Tim Dettmers and Douwe Kiela and Ali Farhadi and Noah A. Smith and Pang Wei Koh and Amanpreet Singh and Hannaneh Hajishirzi},
      year={2024},
      eprint={2409.02060},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2409.02060}, 
}
```