---
title: led-large-16384
canonical_url: "https://www.modelscope.cn/models/LLM-Research/led-large-16384"
md_url: "https://www.modelscope.cn/models/LLM-Research/led-large-16384.md"
repository: LLM-Research/led-large-16384
last_updated: 2024-09-08
license: apache-2.0
pipeline_tag: text2text-generation
tasks:
  - text2text-generation
model_type:
  - led
architectures:
  - LEDForConditionalGeneration
library_name:
  - tensorflow
  - pytorch
frameworks:
  - pytorch
language:
  - en
downloads: 2148
stars: 0
---

# led-large-16384

> led-large-16384 - LLM-Research 在 ModelScope 开源的模型。Allenai's Longformer Encoder-Decoder (LED).

LLM-Research/led-large-16384 是 ModelScope 魔搭社区上的text2text-generation模型，采用 apache-2.0 许可。

- **Repository**: LLM-Research/led-large-16384
- **License**: apache-2.0
- **Tasks**: text2text-generation
- **Downloads**: 2148
- **Stars**: 0
- **Last updated**: 2024-09-08

Source: https://www.modelscope.cn/models/LLM-Research/led-large-16384

---

## Introduction

[Allenai's Longformer Encoder-Decoder (LED)](https://github.com/allenai/longformer#longformer).

As described in [Longformer: The Long-Document Transformer](https://arxiv.org/pdf/2004.05150.pdf) by Iz Beltagy, Matthew E. Peters, Arman Cohan, *led-large-16384* was initialized from [*bart-large*](https://huggingface.co/facebook/bart-large) since both models share the exact same architecture. To be able to process 16K tokens, *bart-large*'s position embedding matrix was simply copied 16 times.

This model is especially interesting for long-range summarization and question answering.

## Fine-tuning for down-stream task

[This notebook](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing) shows how *led-large-16384* can effectively be fine-tuned on a downstream task.