---
title: bilingual-embedding-small
canonical_url: "https://www.modelscope.cn/models/Lajavaness/bilingual-embedding-small"
md_url: "https://www.modelscope.cn/models/Lajavaness/bilingual-embedding-small.md"
repository: Lajavaness/bilingual-embedding-small
last_updated: 2025-07-28
license: apache-2.0
pipeline_tag: sentence-similarity
tasks:
  - sentence-similarity
model_type:
  - bilingual
architectures:
  - BilingualModel
parameters: 117.7M
tensor_type:
  - F32
library_name:
  - sentence-transformers
  - safetensors
  - pytorch
frameworks:
  - pytorch
language:
  - fr
  - en
downloads: 832
stars: 0
tags:
  - sentence-transformers
  - feature-extraction
  - sentence-similarity
  - transformers
  - sentence-embedding
  - mteb
---

# bilingual-embedding-small

> bilingual-embedding-small - Lajavaness 在 ModelScope 开源的模型。bilingual-embedding-small

Lajavaness/bilingual-embedding-small 是 ModelScope 魔搭社区上的 117.7M 参数sentence-similarity模型，采用 apache-2.0 许可。

- **Repository**: Lajavaness/bilingual-embedding-small
- **License**: apache-2.0
- **Tasks**: sentence-similarity
- **Parameters**: 117.7M
- **Tags**: sentence-transformers, feature-extraction, sentence-similarity, transformers, sentence-embedding, mteb
- **Downloads**: 832
- **Stars**: 0
- **Last updated**: 2025-07-28

Source: https://www.modelscope.cn/models/Lajavaness/bilingual-embedding-small

---

# [bilingual-embedding-small](https://huggingface.co/Lajavaness/bilingual-embedding-small)

Bilingual-embedding is the Embedding Model for bilingual language: french and english. This model is a specialized sentence-embedding trained specifically for the bilingual language, leveraging the robust capabilities of [Multilingual-MiniLM-L12-H384](https://huggingface.co/microsoft/Multilingual-MiniLM-L12-H384), a pre-trained language model  is built upon [multilingual-e5](https://huggingface.co/intfloat/multilingual-e5-small) architecture. The model utilizes MiniLM to encode english-french sentences into a 384-dimensional vector space, facilitating a wide range of applications from semantic search to text clustering. The embeddings capture the nuanced meanings of english-french sentences, reflecting both the lexical and contextual layers of the language.


## Full Model Architecture
```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BilingualModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)
```

## Training and Fine-tuning process
#### Stage 1: NLI Training
- Dataset: [(SNLI+XNLI) for english+french]
- Method: Training using Multi-Negative Ranking Loss. This stage focused on improving the model's ability to discern and rank nuanced differences in sentence semantics.
### Stage 3: Continued Fine-tuning for Semantic Textual Similarity on STS Benchmark
- Dataset: [STSB-fr and en]
- Method: Fine-tuning specifically for the semantic textual similarity benchmark using Siamese BERT-Networks configured with the 'sentence-transformers' library. 
### Stage 4: Advanced Augmentation Fine-tuning
- Dataset: STSB with generate [silver sample from gold sample](https://www.sbert.net/examples/training/data_augmentation/README.html)
- Method: Employed an advanced strategy using [Augmented SBERT](https://arxiv.org/abs/2010.08240) with Pair Sampling Strategies, integrating both Cross-Encoder and Bi-Encoder models. This stage further refined the embeddings by enriching the training data dynamically, enhancing the model's robustness and accuracy.


## Usage:

Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:

```
pip install -U sentence-transformers
```

Then you can use the model like this:

```python
from sentence_transformers import SentenceTransformer

sentences = ["Paris est une capitale de la France", "Paris is a capital of France"]

model = SentenceTransformer('Lajavaness/bilingual-embedding-small', trust_remote_code=True)


embeddings = model.encode(sentences)

print(embeddings)

```





## Evaluation

TODO

## Citation

    @article{conneau2019unsupervised,
      title={Unsupervised cross-lingual representation learning at scale},
      author={Conneau, Alexis and Khandelwal, Kartikay and Goyal, Naman and Chaudhary, Vishrav and Wenzek, Guillaume and Guzm{\'a}n, Francisco and Grave, Edouard and Ott, Myle and Zettlemoyer, Luke and Stoyanov, Veselin},
      journal={arXiv preprint arXiv:1911.02116},
      year={2019}
    }

	@article{reimers2019sentence,
	   title={Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
	   author={Nils Reimers, Iryna Gurevych},
	   journal={https://arxiv.org/abs/1908.10084},
	   year={2019}
	}

    @article{thakur2020augmented,
      title={Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks},
      author={Thakur, Nandan and Reimers, Nils and Daxenberger, Johannes and Gurevych, Iryna},
      journal={arXiv e-prints},
      pages={arXiv--2010},
      year={2020}
