---
title: bigbird-base-trivia-itc
canonical_url: "https://www.modelscope.cn/models/google/bigbird-base-trivia-itc"
md_url: "https://www.modelscope.cn/models/google/bigbird-base-trivia-itc.md"
repository: google/bigbird-base-trivia-itc
last_updated: 2025-04-21
license: apache-2.0
pipeline_tag: question-answering
tasks:
  - question-answering
model_type:
  - big_bird
architectures:
  - BigBirdForQuestionAnswering
library_name:
  - transformer
  - pytorch
frameworks:
  - pytorch
language:
  - en
downloads: 428
stars: 0
---

# bigbird-base-trivia-itc

> bigbird-base-trivia-itc - google 在 ModelScope 开源的模型。BigBird base trivia-itc

google/bigbird-base-trivia-itc 是 ModelScope 魔搭社区上的question-answering模型，采用 apache-2.0 许可。

- **Repository**: google/bigbird-base-trivia-itc
- **License**: apache-2.0
- **Tasks**: question-answering
- **Downloads**: 428
- **Stars**: 0
- **Last updated**: 2025-04-21

Source: https://www.modelscope.cn/models/google/bigbird-base-trivia-itc

---

# BigBird base trivia-itc

This model is a fine-tune checkpoint of `bigbird-roberta-base`, fine-tuned on `trivia_qa` with `BigBirdForQuestionAnsweringHead` on its top.

Check out [this](https://colab.research.google.com/drive/1DVOm1VHjW0eKCayFq1N2GpY6GR9M4tJP?usp=sharing) to see how well `google/bigbird-base-trivia-itc` performs on question answering.

## How to use

Here is how to use this model to get the features of a given text in PyTorch:

```python
from transformers import BigBirdForQuestionAnswering

# by default its in `block_sparse` mode with num_random_blocks=3, block_size=64
model = BigBirdForQuestionAnswering.from_pretrained("google/bigbird-base-trivia-itc")

# you can change `attention_type` to full attention like this:
model = BigBirdForQuestionAnswering.from_pretrained("google/bigbird-base-trivia-itc", attention_type="original_full")

# you can change `block_size` & `num_random_blocks` like this:
model = BigBirdForQuestionAnswering.from_pretrained("google/bigbird-base-trivia-itc", block_size=16, num_random_blocks=2)

question = "Replace me by any text you'd like."
context = "Put some context for answering"
encoded_input = tokenizer(question, context, return_tensors='pt')
output = model(**encoded_input)
```

# Fine-tuning config & hyper-parameters

- No. of global token = 128
- Window length = 192
- No. of random token = 192
- Max. sequence length = 4096
- No. of heads = 12
- No. of hidden layers = 12
- Hidden layer size = 768
- Batch size = 32
- Loss = cross-entropy noisy spans

## BibTeX entry and citation info

```tex
@misc{zaheer2021big,
      title={Big Bird: Transformers for Longer Sequences}, 
      author={Manzil Zaheer and Guru Guruganesh and Avinava Dubey and Joshua Ainslie and Chris Alberti and Santiago Ontanon and Philip Pham and Anirudh Ravula and Qifan Wang and Li Yang and Amr Ahmed},
      year={2021},
      eprint={2007.14062},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}
```