--- title: Erlangshen-TCBert-330M-Sentence-Embedding-Chinese canonical_url: "https://www.modelscope.cn/models/Fengshenbang/Erlangshen-TCBert-330M-Sentence-Embedding-Chinese" md_url: "https://www.modelscope.cn/models/Fengshenbang/Erlangshen-TCBert-330M-Sentence-Embedding-Chinese.md" repository: Fengshenbang/Erlangshen-TCBert-330M-Sentence-Embedding-Chinese chinese_name: "二郎神-TCBert-330M-Sentence-Embedding-中文" last_updated: 2023-02-20 license: "Apache License 2.0" pipeline_tag: fill-mask tasks: - fill-mask model_type: - bert architectures: - BertForMaskedLM library_name: - pytorch frameworks: - pytorch language: - zh downloads: 162 stars: 1 tags: - classification --- # Erlangshen-TCBert-330M-Sentence-Embedding-Chinese > Erlangshen-TCBert-330M-Sentence-Embedding-Chinese - Fengshenbang 在 ModelScope 开源的模型。330M参数的句子表征Topic Classification BERT (TCBert)。 Fengshenbang/Erlangshen-TCBert-330M-Sentence-Embedding-Chinese 是 ModelScope 魔搭社区上的fill-mask模型，采用 Apache License 2.0 许可。 - **Repository**: Fengshenbang/Erlangshen-TCBert-330M-Sentence-Embedding-Chinese - **License**: Apache License 2.0 - **Tasks**: fill-mask - **Tags**: classification - **Downloads**: 162 - **Stars**: 1 - **Last updated**: 2023-02-20 Source: https://www.modelscope.cn/models/Fengshenbang/Erlangshen-TCBert-330M-Sentence-Embedding-Chinese --- # IDEA-CCNL/Erlangshen-TCBert-330M-Sentence-Embedding-Chinese - Github: [Fengshenbang-LM](https://github.com/IDEA-CCNL/Fengshenbang-LM) - Docs: [Fengshenbang-Docs](https://fengshenbang-doc.readthedocs.io/) ## 简介 Brief Introduction 330M参数的句子表征Topic Classification BERT (TCBert)。 The TCBert with 330M parameters is pre-trained for sentence representation for Chinese topic classification tasks. ## 模型分类 Model Taxonomy | 需求 Demand | 任务 Task | 系列 Series | 模型 Model | 参数 Parameter | 额外 Extra | | :----: | :----: | :----: | :----: | :----: | :----: | | 通用 General | 句子表征 | 二郎神 Erlangshen | TCBert (sentence representation) | 330M | Chinese | ## 模型信息 Model Information 为了提高模型在话题分类上句子表征效果，我们收集了大量话题分类数据进行基于prompts的对比学习预训练。 To improve the model performance on sentence representation for the topic classification task, we collected numerous topic classification datasets for contrastive pre-training based on general prompts. ### 下游效果 Performance 我们为每个数据集设计了两个prompt模板。 We customize two prompts templates for each dataset. 第一个prompt模板： For ***prompt template 1***: | Dataset | Prompt template 1 | |---------|:------------------------:| | TNEWS | 下面是一则关于__的新闻： | | CSLDCP | 这一句描述__的内容如下： | | IFLYTEK | 这一句描述__的内容如下： | 第一个prompt模板的微调实验结果： The **fine-tuning** results for prompt template 1: | Model | TNEWS | CLSDCP | IFLYTEK | |-----------------|:------:|:------:|:-------:| | Macbert-base | 55.02 | 57.37 | 51.34 | | Macbert-large | 55.77 | 58.99 | 50.31 | | Erlangshen-1.3B | 57.36 | 62.35 | 53.23 | | TCBert-base110M-Classification-Chinese | 55.57 | 58.60 | 49.63 | | TCBert-large330M-Classification-Chinese | 56.17 | 60.06 | 51.34 | | TCBert-1.3B1.3B-Classification-Chinese | 57.41 | 65.10 | 53.75 | | TCBert-base110M-Sentence-Embedding-Chinese | 54.68 | 59.78 | 49.40 | | TCBert-large330M-Sentence-Embedding-Chinese | 55.32 | 62.07 | 51.11 | | TCBert-1.3B1.3B-Sentence-Embedding-Chinese | 57.46 | 65.04 | 53.06 | 第一个prompt模板的句子相似度结果： The **sentence similarity** results for prompt template 1: | | TNEWS | | CSLDCP | | IFLYTEK | | |-----------------|:--------:|:---------:|:---------:|:---------:|:---------:|:---------:| | Model | referece | whitening | reference | whitening | reference | whitening | | Macbert-base | 43.53 | 47.16 | 33.50 | 36.53 | 28.99 | 33.85 | | Macbert-large | 46.17 | 49.35 | 37.65 | 39.38 | 32.36 | 35.33 | | Erlangshen-1.3B | 45.72 | 49.60 | 40.56 | 44.26 | 29.33 | 36.48 | | TCBert-base110M-Classification-Chinese | 48.61 | 51.99 | 43.31 | 45.15 | 33.45 | 37.28 | | TCBert-large330M-Classification-Chinese | 50.50 | 52.79 | 52.89 | 53.89 | 34.93 | 38.31 | | TCBert-1.3B1.3B-Classification-Chinese | 50.80 | 51.59 | 51.93 | 54.12 | 33.96 | 38.08 | | TCBert-base110M-Sentence-Embedding-Chinese | 45.82 | 47.06 | 42.91 | 43.87 | 33.28 | 34.76 | | TCBert-large330M-Sentence-Embedding-Chinese | 50.10 | 50.90 | 53.78 | 53.33 | 37.62 | 36.94 | | TCBert-1.3B1.3B-Sentence-Embedding-Chinese | 50.70 | 53.48 | 52.66 | 54.40 | 36.88 | 38.48 | 第二个prompt模板： For ***prompt template 2***: | Dataset | Prompt template 2 | |---------|:------------------------:| | TNEWS | 接下来的新闻，是跟__相关的内容： | | CSLDCP | 接下来的学科，是跟__相关： | | IFLYTEK | 接下来的生活内容，是跟__相关： | 第二个prompt模板的微调结果： The **fine-tuning** results for prompt template 2: | Model | TNEWS | CLSDCP | IFLYTEK | |-----------------|:------:|:------:|:-------:| | Macbert-base | 54.78 | 58.38 | 50.83 | | Macbert-large | 56.77 | 60.22 | 51.63 | | Erlangshen-1.3B | 57.81 | 62.80 | 52.77 | | TCBert-base110M-Classification-Chinese | 54.58 | 59.16 | 49.80 | | TCBert-large330M-Classification-Chinese | 56.22 | 61.23 | 50.77 | | TCBert-1.3B1.3B-Classification-Chinese | 57.41 | 64.82 | 53.34 | | TCBert-base110M-Sentence-Embedding-Chinese | 54.68 | 59.78 | 49.40 | | TCBert-large330M-Sentence-Embedding-Chinese | 55.32 | 62.07 | 51.11 | | TCBert-1.3B1.3B-Sentence-Embedding-Chinese | 56.87 | 65.83 | 52.94 | 第二个prompt模板的句子相似度结果： The **sentence similarity** results for prompt template 2: | | TNEWS | | CSLDCP | | IFLYTEK | | |-----------------|:--------:|:---------:|:---------:|:---------:|:---------:|:---------:| | Model | referece | whitening | reference | whitening | reference | whitening | | Macbert-base | 42.29 | 45.22 | 34.23 | 37.48 | 29.62 | 34.13 | | Macbert-large | 46.22 | 49.60 | 40.11 | 44.26 | 32.36 | 35.16 | | Erlangshen-1.3B | 46.17 | 49.10 | 40.45 | 45.88 | 30.36 | 36.88 | | TCBert-base110M-Classification-Chinese | 48.31 | 51.34 | 43.42 | 45.27 | 33.10 | 36.19 | | TCBert-large330M-Classification-Chinese | 51.19 | 51.69 | 52.55 | 53.28 | 34.31 | 37.45 | | TCBert-1.3B1.3B-Classification-Chinese | 52.14 | 52.39 | 51.71 | 53.89 | 33.62 | 38.14 | | TCBert-base110M-Sentence-Embedding-Chinese | 46.72 | 48.86 | 43.19 | 43.53 | 34.08 | 35.79 | | TCBert-large330M-Sentence-Embedding-Chinese | 50.65 | 51.94 | 53.84 | 53.67 | 37.74 | 36.65 | | TCBert-1.3B1.3B-Sentence-Embedding-Chinese | 50.75 | 54.78 | 51.43 | 54.34 | 36.48 | 38.36 | 更多关于TCBERTs的细节，请参考我们的技术报告。基于新的数据，我们会更新TCBERTs，请留意我们仓库的更新。 For more details about TCBERTs, please refer to our paper. We may regularly update TCBERTs upon new coming data, please keep an eye on the repo! ## 使用 Usage ```python from modelscope.pipelines import pipeline from modelscope.utils.constant import Tasks pipeline_ins = pipeline( 'fill-mask', model='Fengshenbang/Erlangshen-TCBert-330M-Sentence-Embedding-Chinese', model_revision='v1.0.1' ) print(pipeline_ins('中国首都位于')) ``` ## 引用 Citation 如果您在您的工作中使用了我们的模型，可以引用我们的[技术报告](https://arxiv.org/abs/2211.11304): If you use for your work, please cite the following paper ``` @article{han2022tcbert, title={TCBERT: A Technical Report for Chinese Topic Classification BERT}, author={Han, Ting and Pan, Kunhao and Chen, Xinyu and Song, Dingjie and Fan, Yuchen and Gao, Xinyu and Gan, Ruyi and Zhang, Jiaxing}, journal={arXiv preprint arXiv:2211.11304}, year={2022} } ``` 如果您在您的工作中使用了我们的模型，可以引用我们的[网站](https://github.com/IDEA-CCNL/Fengshenbang-LM/): You can also cite our [website](https://github.com/IDEA-CCNL/Fengshenbang-LM/): ```text @misc{Fengshenbang-LM, title={Fengshenbang-LM}, author={IDEA-CCNL}, year={2021}, howpublished={\url{https://github.com/IDEA-CCNL/Fengshenbang-LM}}, } ```