---
title: plant-dnagpt-6mer
canonical_url: "https://www.modelscope.cn/models/zhangtaolab/plant-dnagpt-6mer"
md_url: "https://www.modelscope.cn/models/zhangtaolab/plant-dnagpt-6mer.md"
repository: zhangtaolab/plant-dnagpt-6mer
last_updated: 2024-10-16
license: CC-BY-NC-SA-4.0
pipeline_tag: fill-mask
tasks:
  - fill-mask
model_type:
  - GPT-2
architectures:
  - GPT2LMHeadModel
parameters: 89.0M
tensor_type:
  - F32
library_name:
  - pytorch
  - transformer
  - safetensors
frameworks:
  - Pytorch
inference_backends:
  - "deploy_task text/emb"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 178
stars: 0
tags:
  - DNA
  - biology
  - genomics
---

# plant-dnagpt-6mer

> plant-dnagpt-6mer - zhangtaolab 在 ModelScope 开源的模型。植物基础DNA大语言模型 (Plant foundation DNA large language models)

zhangtaolab/plant-dnagpt-6mer 是 ModelScope 魔搭社区上的 89.0M 参数fill-mask模型，采用 CC-BY-NC-SA-4.0 许可，可用 deploy_task text/emb、sglang 0.5.2、vllm 0.9.2 部署。

- **Repository**: zhangtaolab/plant-dnagpt-6mer
- **License**: CC-BY-NC-SA-4.0
- **Tasks**: fill-mask
- **Parameters**: 89.0M
- **Inference backends**: deploy_task text/emb, sglang 0.5.2, vllm 0.9.2
- **Tags**: DNA, biology, genomics
- **Downloads**: 178
- **Stars**: 0
- **Last updated**: 2024-10-16

Source: https://www.modelscope.cn/models/zhangtaolab/plant-dnagpt-6mer

---

# 植物基础DNA大语言模型 (Plant foundation DNA large language models)

The plant DNA large language models (LLMs) contain a series of foundation models based on different model architectures, which are pre-trained on various plant reference genomes.  
All the models have a comparable model size between 90 MB and 150 MB, BPE tokenizer is used for tokenization and 8000 tokens are included in the vocabulary.  

**开发者:** zhangtaolab


### Model Sources

- **Repository:** [Plant DNA LLMs](https://github.com/zhangtaolab/plant_DNA_LLMs)
- **Manuscript:** [Versatile applications of foundation DNA large language models in plant genomes]()

### Architecture

The model is trained based on the OpenAI GPT-2 model with modified tokenizer specific for DNA sequence.

### How to use

Install the runtime library first:
```bash
pip install transformers
```

Here is a simple code for inference:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = 'plant-dnagpt-6mer'
# load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(f'zhangtaolab/{model_name}', trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(f'zhangtaolab/{model_name}', trust_remote_code=True)

# example sequence and tokenization
sequences = ['ATATACGGCCGNC','GGGTATCGCTTCCGAC']
tokens = tokenizer(sequences,padding="longest")['input_ids']
print(f"Tokenzied sequence: {tokenizer.batch_decode(tokens)}")

# inference
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
model.to(device)
inputs = tokenizer(sequences, truncation=True, padding='max_length', max_length=512, 
                   return_tensors="pt")
inputs = {k: v.to(device) for k, v in inputs.items()}
outs = model(
    **inputs,
    output_hidden_states=True
)

# get the final layer embeddings and prediction logits
embeddings = outs['hidden_states'][-1].detach().numpy()
logits = outs['logits'].detach().numpy()
```


### Training data
We use CausalLM method to pre-train the model, the tokenized sequence have a maximum length of 512.  
Detailed training procedure can be found in our manuscript.


#### Hardware
Model was pre-trained on a NVIDIA RTX4090 GPU (24 GB).
