---
title: scibert_scivocab_cased
canonical_url: "https://www.modelscope.cn/models/LLM-Research/scibert_scivocab_cased"
md_url: "https://www.modelscope.cn/models/LLM-Research/scibert_scivocab_cased.md"
repository: LLM-Research/scibert_scivocab_cased
last_updated: 2024-09-08
model_type:
  - bert
library_name:
  - pytorch
frameworks:
  - pytorch
language:
  - en
downloads: 2141
stars: 0
---

# scibert_scivocab_cased

> scibert_scivocab_cased - LLM-Research 在 ModelScope 开源的模型。This is the pretrained model presented in SciBERT: A Pretrained Language Model for Scientific Text, which is a BERT model trained on scientific text.

- **Repository**: LLM-Research/scibert_scivocab_cased
- **Downloads**: 2141
- **Stars**: 0
- **Last updated**: 2024-09-08

Source: https://www.modelscope.cn/models/LLM-Research/scibert_scivocab_cased

---

# SciBERT

This is the pretrained model presented in [SciBERT: A Pretrained Language Model for Scientific Text](https://www.aclweb.org/anthology/D19-1371/), which is a BERT model trained on scientific text.

The training corpus was papers taken from [Semantic Scholar](https://www.semanticscholar.org). Corpus size is 1.14M papers, 3.1B tokens. We use the full text of the papers in training, not just abstracts.

SciBERT has its own wordpiece vocabulary (scivocab) that's built to best match the training corpus. We trained cased and uncased versions. 

Available models include:
* `scibert_scivocab_cased`
* `scibert_scivocab_uncased`


The original repo can be found [here](https://github.com/allenai/scibert).

If using these models, please cite the following paper:
```
@inproceedings{beltagy-etal-2019-scibert,
    title = "SciBERT: A Pretrained Language Model for Scientific Text",
    author = "Beltagy, Iz  and Lo, Kyle  and Cohan, Arman",
    booktitle = "EMNLP",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/D19-1371"
}
```