---
title: blt-7b
canonical_url: "https://www.modelscope.cn/models/facebook/blt-7b"
md_url: "https://www.modelscope.cn/models/facebook/blt-7b.md"
repository: facebook/blt-7b
last_updated: 2025-05-20
license: other
parameters: 10.6B
tensor_type:
  - BF16
library_name:
  - safetensors
  - pytorch
frameworks:
  - pytorch
language:
  - en
downloads: 339
stars: 0
tags:
  - facebook
  - meta-pytorch
  - blt
---

# blt-7b

> blt-7b - facebook 在 ModelScope 开源的模型。Byte Latent Transformer (BLT)

facebook/blt-7b 是 ModelScope 魔搭社区上的 10.6B 参数机器学习模型，采用 other 许可。

- **Repository**: facebook/blt-7b
- **License**: other
- **Parameters**: 10.6B
- **Tags**: facebook, meta-pytorch, blt
- **Downloads**: 339
- **Stars**: 0
- **Last updated**: 2025-05-20

Source: https://www.modelscope.cn/models/facebook/blt-7b

---

# Byte Latent Transformer (BLT)

This repository contains the model weights for our paper: "Byte Latent Transformer: Patches Scale Better Than Tokens"

- [Paper Link](https://dl.fbaipublicfiles.com/blt/BLT__Patches_Scale_Better_Than_Tokens.pdf)
- [HF Paper Link](https://huggingface.co/papers/2412.09871)

## Abstract

We introduce the Byte Latent Transformer architecture (BLTs), a new byte-level LLM architecture that
for the first time, matches tokenization-based LLM performance at scale, with significant improvements
in inference efficiency and robustness. BLT encodes bytes into dynamically sized patches, which serve
as the primary units of computation. Patches are segmented dynamically based on the entropy of the
next byte, allocating more compute and model capacity where there is more data complexity. The BLT
architecture includes new attention mechanisms to maximize the information flow between byte and
patch hidden representations and a new type of byte-sequence memory. We present the first scaling
study of byte-level models up to 8B parameters and 8T training bytes, showing for the first time
that we can train a model end-to-end at scale from bytes with no tokenization or other preprocessing.
Scaling trends reveal training and inference efficiency benefits from dynamically selecting very long
patches on average, along with qualitative improvements with reasoning and long tail generalization
from modeling byte-sequences.

To run the model, see the readme here: https://github.com/facebookresearch/blt

## Links

- Code: https://github.com/facebookresearch/blt
- BLT 1B Weights: https://huggingface.co/facebook/blt-1b
- BLT 7B Weights: https://huggingface.co/facebook/blt-7b
- BLT Weight Collection: https://huggingface.co/collections/facebook/blt-6801263d4ac1704702a192a6