---
title: PEG
canonical_url: "https://www.modelscope.cn/models/AI-ModelScope/PEG"
md_url: "https://www.modelscope.cn/models/AI-ModelScope/PEG.md"
repository: AI-ModelScope/PEG
last_updated: 2024-01-16
license: apache-2.0
pipeline_tag: sentence-similarity
tasks:
  - sentence-similarity
model_type:
  - bert
architectures:
  - BertModel
library_name:
  - pytorch
frameworks:
  - Pytorch
language:
  - zh
inference_backends:
  - "deploy_task emb"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 41052
stars: 0
tags:
  - PEG
  - feature-extraction
  - sentence-similarity
  - transformers
  - mteb
---

# PEG

> PEG - AI-ModelScope 在 ModelScope 开源的模型。PEG: Towards Robust Text Retrieval with Progressive Learning

AI-ModelScope/PEG 是 ModelScope 魔搭社区上的sentence-similarity模型，采用 apache-2.0 许可，可用 deploy_task emb、sglang 0.5.2、vllm 0.9.2 部署。

- **Repository**: AI-ModelScope/PEG
- **License**: apache-2.0
- **Tasks**: sentence-similarity
- **Inference backends**: deploy_task emb, sglang 0.5.2, vllm 0.9.2
- **Tags**: PEG, feature-extraction, sentence-similarity, transformers, mteb
- **Downloads**: 41052
- **Stars**: 0
- **Last updated**: 2024-01-16

Source: https://www.modelscope.cn/models/AI-ModelScope/PEG

---

<h1 align="center">PEG: Towards Robust Text Retrieval with Progressive Learning</h1>

## Model Details
We propose the PEG model (a Progressively Learned Textual Embedding), which progressively adjusts the weights of samples contributing to the loss within an extremely large batch, based on the difficulty levels of negative samples.
we have amassed an extensive collection of over 110 million data, spanning a wide range of fields such as general knowledge, finance, tourism, medicine, and more.

Our technical report is available at [Paper](https://arxiv.org/pdf/2311.11691.pdf)

## Usage 

Install modelscope:
```
pip install modelscope
```

Then load model and predict:
```python
from modelscope import AutoModel, AutoTokenizer
import torch


# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('TownsWu/PEG')
model = AutoModel.from_pretrained('TownsWu/PEG')
sentences = ['如何更换花呗绑定银行卡', '花呗更改绑定银行卡']
# Tokenize sentences
inputs = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

# Compute token embeddings
with torch.no_grad():
    last_hidden_state = model(**inputs, return_dict=True).last_hidden_state
    embeddings = last_hidden_state[:, 0]
print("embeddings:")
print(embeddings)
```

## Contact
If you have any question or suggestion related to this project, feel free to open an issue or pull request.
You also can email Tong Wu(townswu@tencent.com). 


## Citation

If you find our work helpful for your research, please consider citing the following BibTeX entry:

```

@article{wu2023towards,
  title={Towards Robust Text Retrieval with Progressive Learning},
  author={Wu, Tong and Qin, Yulei and Zhang, Enwei and Xu, Zihan and Gao, Yuting and Li, Ke and Sun, Xing},
  journal={arXiv preprint arXiv:2311.11691},
  year={2023}
}

```