---
title: llama-3-tulu-2-8b-uf-mean-rm
canonical_url: "https://www.modelscope.cn/models/LLM-Research/llama-3-tulu-2-8b-uf-mean-rm"
md_url: "https://www.modelscope.cn/models/LLM-Research/llama-3-tulu-2-8b-uf-mean-rm.md"
repository: LLM-Research/llama-3-tulu-2-8b-uf-mean-rm
last_updated: 2024-09-07
license: apache-2.0
pipeline_tag: text-classification
tasks:
  - text-classification
model_type:
  - llama
architectures:
  - LlamaForSequenceClassification
base_model:
  - allenai/llama-3-tulu-2-8b
base_model_relation: finetune
parameters: 7.5B
tensor_type:
  - BF16
library_name:
  - safetensors
  - pytorch
frameworks:
  - pytorch
language:
  - en
inference_backends:
  - "deploy_task emb"
  - "sglang 0.5.2"
downloads: 2178
stars: 0
---

# llama-3-tulu-2-8b-uf-mean-rm

> llama-3-tulu-2-8b-uf-mean-rm - LLM-Research 在 ModelScope 开源的模型。Model Card for Llama 3 Tulu V2 8B RM - UltraFeedback

LLM-Research/llama-3-tulu-2-8b-uf-mean-rm 是 ModelScope 魔搭社区上的 7.5B 参数text-classification模型，采用 apache-2.0 许可，基于 allenai/llama-3-tulu-2-8b 构建，可用 deploy_task emb、sglang 0.5.2 部署。

- **Repository**: LLM-Research/llama-3-tulu-2-8b-uf-mean-rm
- **License**: apache-2.0
- **Tasks**: text-classification
- **Parameters**: 7.5B
- **Base model**: allenai/llama-3-tulu-2-8b
- **Inference backends**: deploy_task emb, sglang 0.5.2
- **Downloads**: 2178
- **Stars**: 0
- **Last updated**: 2024-09-07

Source: https://www.modelscope.cn/models/LLM-Research/llama-3-tulu-2-8b-uf-mean-rm

---

<center>
<img src="https://huggingface.co/datasets/allenai/blog-images/resolve/main/tulu-2.5/tulu_25_banner.png" alt="Tulu 2.5 banner image" width="800px"/>
</center>

# Model Card for Llama 3 Tulu V2 8B RM - UltraFeedback

Tulu is a series of language models that are trained to act as helpful assistants.
This is a 8B reward model used for PPO training trained on the UltraFeedback dataset.

For more details, read the paper:
[Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback](https://arxiv.org/abs/2406.09279).

**Built with Meta Llama 3!**
Note that Llama 3 is released under the Meta Llama 3 community license, included here under `llama_3_license.txt`.

## Performance

We evaluate the model on [RewardBench](https://github.com/allenai/reward-bench):

| Model            | Score | Chat  | Chat Hard | Safety | Reasoning |
|------------------|-------|-------|-----------|--------|-----------|
| **[Llama 3 Tulu 2 8b UF RM](https://huggingface.co/allenai/llama-3-tulu-2-8b-uf-mean-rm) (this model)**  | 73.6  | 95.3  |    59.2   |  57.9  |    82.1   |
| [Llama 3 Tulu 2 70b UF RM](https://huggingface.co/allenai/llama-3-tulu-2-70b-uf-mean-rm) |    71.0 | 86.3 | 56.1 | 58.9 | 82.7 |


## Model description

- **Model type:** A reward model trained on UltraFeedback, designed to be used in RLHF training.
- **Language(s) (NLP):** English
- **License:** Apache 2.0.
- **Finetuned from model:** [allenai/llama-3-tulu-2-8b](https://huggingface.co/allenai/llama-3-tulu-2-8b)

### Model Sources

- **Repository:** https://github.com/allenai/open-instruct
- **Dataset:** Data used to train this model can be found [here](https://huggingface.co/datasets/allenai/tulu-2.5-preference-data) - specifically the `ultrafeedback_mean_aspects` split.


## Input Format

The model is trained to use the following format (note the newlines):
```
<|user|>
Your message here!
<|assistant|>
```

For best results, format all inputs in this manner. **Make sure to include a newline after `<|assistant|>`, this can affect generation quality quite a bit.**
We have included a [chat template](https://huggingface.co/docs/transformers/main/en/chat_templating) in the tokenizer implementing this template.

## Intended uses & limitations

The model was initially fine-tuned on a filtered and preprocessed of the [Tulu V2 mix dataset](https://huggingface.co/datasets/allenai/tulu-v2-sft-mixture), which contains a diverse range of human created instructions and synthetic dialogues generated primarily by other LLMs. 
We then further trained the model with a [Jax RM trainer](https://github.com/hamishivi/EasyLM/blob/main/EasyLM/models/llama/llama_train_rm.py) built on [EasyLM](https://github.com/young-geng/EasyLM) on the dataset mentioned above.
This model is meant as a research artefact.

### Training hyperparameters

The following hyperparameters were used during PPO training:
- learning_rate: 1e-06
- total_train_batch_size: 512
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear cooldown to 1e-05.
- lr_scheduler_warmup_ratio: 0.03
- num_epochs: 1.0

## Citation

If you find Tulu 2.5 is useful in your work, please cite it with:

```
@misc{ivison2024unpacking,
      title={{Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback}}, 
      author={{Hamish Ivison and Yizhong Wang and Jiacheng Liu and Ellen Wu and Valentina Pyatkin and Nathan Lambert and Yejin Choi and Noah A. Smith and Hannaneh Hajishirzi}}
      year={2024},
      eprint={2406.09279},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```