---
title: arxiv-complete
canonical_url: "https://www.modelscope.cn/datasets/secemp9/arxiv-complete"
md_url: "https://www.modelscope.cn/datasets/secemp9/arxiv-complete.md"
repository: secemp9/arxiv-complete
last_updated: 2026-09-21
license: other
storage_size: "15 TB"
downloads: 431209
stars: 44
---
# arxiv-complete
> arxiv-complete - secemp9 在 ModelScope 开源的数据集。arXiv Complete Corpus
secemp9/arxiv-complete 是 ModelScope 魔搭社区上的数据集，存储大小 15 TB，采用 other 许可。
- **Repository**: secemp9/arxiv-complete
- **License**: other
- **Storage size**: 15 TB
- **Downloads**: 431209
- **Stars**: 44
- **Last updated**: 2026-09-21
Source: https://www.modelscope.cn/datasets/secemp9/arxiv-complete
---
# arXiv Complete Corpus
A snapshot of arXiv's metadata, version history, submission files and rendered
documents. It covers 3,148,796 papers and includes file contents, paths, sizes
and SHA-256 digests. Metadata comes from arXiv's OAI-PMH `arXivRaw` interface;
files come from the GCS mirror, S3 source archives and direct PDF fetches.
This release holds a PDF for 99.47% of papers and 99.54% of versions reported
with a non-zero submission size. It is a one-off snapshot; coverage gaps are
documented below.
## Start here
Start with the 26 MB sample:
```python
from datasets import load_dataset
ds = load_dataset("secemp9/arxiv-complete", "sample", split="train")
paper = ds[0]
print(paper["paper_id"], paper["title"], len(paper["text"]))
```
Choose `paper_text` for one resolved TeX string per paper, `latex` for the
individual TeX-related files, `source` for complete submission packages, or
`pdf` for rendered documents. Use `metadata`, `versions` and `files` to plan a
selection before fetching content.
The main limits to account for are:
- `paper_text` retains LaTeX syntax, comments and macros. It includes
withdrawal stubs and some incorrectly selected templates; it needs filtering
before use as a training corpus.
- Source packages are unversioned. A paper's newest held PDF can be older than
the newest version arXiv reports.
- Large binary rows and row groups require substantial memory. Early exit
from `datasets` streams also failed on the versions tested for this release.
- The `html` family is indexed in `files` but has no content config.
- Licences vary by paper. The compilation's CC0 dedication does not apply to
the papers; see [LICENSE](LICENSE).
[Coverage](#snapshot-and-coverage) · [Reading data](#reading-data) ·
[`paper_text`](#paper-text) · [`source`](#source-content) ·
[`latex` and duplication](#latex-and-duplication) · [Schema](#schema) ·
[Provenance and verification](#provenance-and-verification) ·
[File quality](#file-quality-and-omissions)
## Configs
| config | one row is | rows | papers | of 3,148,796 | Parquet size |
|---|---|---|---|---|---|
| `sample` | a paper sampled from `paper_text` | 991 | 991 | — | 26 MB |
| `metadata` | a paper | 3,148,796 | 3,148,796 | 100% | 1.6 GB |
| `versions` | a reported (paper, version) | 5,030,612 | 3,148,796 | 100% | 269 MB |
| `files` | an indexed file, without content | 54,597,326 | 3,134,898 | 99.56% | 2.4 GB |
| `paper_text` | a paper's resolved TeX | 2,856,227 | 2,856,227 | 90.71% | 70 GB |
| `latex` | a TeX-related file, with content | 12,348,082 | 2,860,286 | 90.84% | 0.16 TB |
| `source` | a submission file, with content | 47,373,604 | 3,121,338 | 99.13% | 6.51 TB |
| `pdf` | an object from the PDF mirror tree | 4,973,956 | 3,131,959 | 99.47% | 8.65 TB |
| `ps` | an object from the PostScript mirror tree | 2,237,839 | 1,318,462 | 41.87% | 0.68 TB |
Parquet sizes are compressed bytes on disk, in decimal SI units
(1 TB = 1,000,000,000,000 bytes). All nine configs total
16,076,056,758,128 bytes, or 16.08 TB. Elsewhere, raw content bytes means
`sum(size)` over the relevant files, before Parquet compression. For example,
`pdf` contains 10.37 TB of raw content in 8.65 TB of Parquet; `source` contains
9.52 TB in 6.51 TB. Sizes and percentages below identify which basis they use.
`papers` is `count(DISTINCT paper_id)`. All counts cover every shard of the
named config. No config contains a paper absent from `metadata`, and
`paper_id` is never null. `sample` is drawn from `paper_text` to demonstrate
the schema. Use the full configs for coverage counts.
```sql
SELECT count(*) AS rows, count(DISTINCT paper_id) AS papers
FROM 'hf://datasets/secemp9/arxiv-complete/ps/*.parquet';
-- 2,237,839 rows; 1,318,462 papers
```
The author-supplied configs overlap: `latex` is a subset of `source`, and
`paper_text` is assembled from its `.tex` and `.ltx` files plus formatted
bibliographies. `pdf` and `ps` contain rendered documents and, in some cases,
additional assets. PostScript represents rendered pages; LaTeX is source code.
### Download one config
An unfiltered `snapshot_download()` attempts to download all 16.08 TB. Use
`allow_patterns` to select a config:
```python
from huggingface_hub import snapshot_download
snapshot_download(
"secemp9/arxiv-complete",
repo_type="dataset",
allow_patterns="paper_text/*",
)
```
The three index configs total 4,345,782,824 Parquet bytes (4.35 GB).
`files` describes 22,571,129,586,192 raw content bytes, including the indexed
but unpublished `html` family. Remote SQL queries can project columns and
prune row groups without downloading whole configs. They still transfer the
metadata and column data needed by the query.
## Snapshot and coverage
| item | value |
|---|---|
| metadata harvested | 2026-08-30, OAI-PMH `arXivRaw` |
| file mirror taken | 2026-09-05 |
| papers | 3,148,796 |
| earliest submission date | 1986-04-25; an original preprint date predating arXiv |
| latest submission date | 2026-08-27 |
There are 353 papers dated 1991 and 284,162 dated 2025. The largest primary
categories are `cs.CV` (5.0%), `hep-ph` (4.6%), `cs.LG` (4.5%), `quant-ph`
(4.3%), `hep-th` (3.6%), `astro-ph` (3.0%) and `cs.CL` (2.8%).
For 67 papers, `first_version_date` preserves an original preprint date that
predates the month in `paper_id`.
### Version coverage
Of the 4,996,775 versions arXiv reports with non-zero submission size,
4,973,594 (99.54%) have a PDF here. The table uses *retrievable* as shorthand
for a non-zero reported size; it does not guarantee a live download exists.
| population | versions | share |
|---|---|---|
| all reported (paper, version) pairs | 5,030,612 | |
| metadata-only: reported size `0kb` | 33,837 | 0.67% of all versions |
| retrievable: reported size above zero | 4,996,775 | 99.33% of all versions |
| retrievable, with PDF held | 4,973,594 | 99.54% of retrievable versions |
| retrievable, with PDF, PostScript or HTML held | 4,975,584 | 99.58% of retrievable versions |
| retrievable, with none of those artifacts held | 21,191 | 0.42% of retrievable versions |
```sql
SELECT count(*) AS versions,
count(*) FILTER (WHERE is_metadata_only) AS metadata_only,
count(*) FILTER (WHERE NOT is_metadata_only) AS retrievable,
count(*) FILTER (WHERE NOT is_metadata_only AND has_pdf) AS with_pdf,
count(*) FILTER (WHERE NOT is_metadata_only
AND (has_pdf OR has_ps OR has_html)) AS with_any,
count(*) FILTER (WHERE NOT is_metadata_only AND NOT has_pdf
AND NOT has_ps AND NOT has_html) AS real_gaps,
count(*) FILTER (WHERE NOT has_pdf AND NOT has_ps AND NOT has_html)
AS no_artifact
FROM 'hf://datasets/secemp9/arxiv-complete/versions/*.parquet';
```
`has_pdf` alone is true for 4,973,947 versions, including 353 marked
metadata-only. That is 98.87% of all 5,030,612 reported versions. The
99.54% figure excludes metadata-only rows from both numerator and denominator.
At paper level, 3,131,959 of 3,148,796 papers (99.47%) have at least one PDF;
3,133,383 (99.51%) have a PDF, PostScript or HTML artifact indexed.
`is_metadata_only` is true exactly when arXiv's `` is `0kb`, parsed as
`reported_size_kb = 0`. Neither column is null. These versions affect 29,221
papers and are the newest version of 16,490 of them. They often correspond to
withdrawals. The flag records arXiv's reported size; 509 metadata-only versions
still have an artifact here (330 PDF and PostScript, 156 HTML,
23 PDF alone). Consequently, the 54,519 versions with no artifact consist of
21,191 non-zero-size gaps and 33,328 metadata-only versions.
On 2026-09-13, live `arXivRaw` records for 40 papers covering 122 versions
agreed with every size flag. A separate check of 20 metadata-only PDF URLs
returned 20 HTTP 404s; all 10 held-version controls returned PDFs. These
checks establish agreement only for the versions tested.
Most non-zero-size gaps fall near the snapshot boundary:
| version date | retrievable | with PDF | PDF share | gaps with no artifact |
|---|---|---|---|---|
| through 2026-08-13 | 4,975,606 | 4,970,942 | 99.91% | 2,674 |
| 2026-08-14 onward | 21,169 | 2,652 | 12.53% | 18,517 |
Daily PDF coverage was 97–99% through 2026-08-12, 81.7% on August 13 and
11–15% thereafter. All 20 sampled snapshot-edge URLs returned PDFs on
2026-09-13. Of the 2,674 older gaps, 1,405 date to 1991–1995; 15 sampled
URLs from that group returned 404. The two groups need different treatment.
`versions` indexes what arXiv reports; `files` indexes what the build held.
Every true `has_pdf`, `has_ps` and `has_html` flag has a matching `files` row.
The PDF and PostScript pair counts agree exactly at 4,973,947 and 1,953,094.
In the other direction, `files` contains 36 HTML (paper, version) pairs across
21 papers that arXiv's OAI record does not report, including `1412.7030` v111
and `1611.09139` v70. See [the HTML omission](#html-omission).
### PDF provenance and the gap-filling pass
| `source` | PDF rows | papers | raw content bytes |
|---|---|---|---|
| `gcs` | 4,967,481 | 3,129,694 | 10,357,939,507,786 |
| `web_scrape` | 6,475 | 5,875 | 13,993,491,538 |
The 6,475 `web_scrape` PDFs were fetched from versioned arxiv.org URLs on
2026-08-30. They use the mirror's path layout and are scattered across 1,218
of 1,886 PDF shards. `stored_as`, `stored_size` and `stored_sha256` are null
for both provenances; `source` distinguishes them. Every fetched row's digest
agrees with `versions.pdf_sha256`.
Fetched PDFs range from 31,284 bytes to 129.8 MB (`2503.01938v1`), with a
median of 847 KB versus 636 KB for `gcs`; two exceed 64 MB. All 2,652 held
PDFs for versions dated August 14 onward came from this pass. Across the
corpus, 2,146 papers and 6,266 version pairs have no rendered artifact except
one of these fetched PDFs. A `gcs` row may hold an older rendering;
`web_scrape` records what arxiv.org served on the fetch date.
```sql
SELECT source, count(*), count(DISTINCT paper_id), sum(size)
FROM 'hf://datasets/secemp9/arxiv-complete/files/*.parquet'
WHERE kind = 'pdf'
GROUP BY source;
```
#### Requests and outcomes
The pass targeted 10,085 version pairs missing a PDF on 2026-08-30. Each
was requested from `/pdf/`, the path arXiv's `robots.txt` explicitly allows,
from a single host behind a global rate limiter that doubles its interval on any
throttle signal. **Only the first 289 went at the fifteen-second `Crawl-delay`
that file declares; the remaining 9,796 went at a 1.20-second global interval —
0.83 requests a second, 12.5× the rate arXiv publishes.** arXiv signalled
throttling zero times in 10,224 requests, but the faster rate was our choice
and it is recorded here rather than left out. Every non-success was requested
a second time. The final tally, one row per target:
| outcome | pairs | response |
|---|---|---|
| recovered | 6,475 | PDF, 13,993,491,538 bytes in total |
| HTTP 404 | 3,471 | 512-byte HTML not-found page |
| HTTP 200, empty body | 91 | `Content-Type: application/pdf`, zero bytes |
| HTTP 500 | 48 | server error repeated on retry |
| version date | requested | recovered | recovery rate |
|---|---|---|---|
| 1991–1995 | 1,443 | 0 | 0.0% |
| 1996–2000 | 561 | 9 | 1.6% |
| 2001–2010 | 811 | 341 | 42.0% |
| 2011–2020 | 3,704 | 2,886 | 77.9% |
| 2021–2026 | 3,566 | 3,239 | 90.8% |
The zero recovery rate for 1991–1995 applies to the missing-PDF request list.
The dataset already holds PDFs for 30,898 of that era's 32,341 retrievable
versions (95.5%); for 1993 alone, it holds 5,985 of 6,530 (91.7%).
#### Remaining gaps
| version date | gaps | requested in the pass | 404 | empty 200 | 500 | never requested |
|---|---|---|---|---|---|---|
| 1991–1995 | 1,405 | 1,405 | 1,404 | 0 | 1 | 0 |
| 1996–2000 | 14 | 14 | 14 | 0 | 0 | 0 |
| 2001–2010 | 40 | 40 | 29 | 7 | 4 | 0 |
| 2011–2020 | 124 | 124 | 95 | 6 | 23 | 0 |
| 2021–2025 | 57 | 57 | 30 | 10 | 17 | 0 |
| 2026-01-01 through 08-13 | 1,034 | 2 | 0 | 0 | 2 | 1,032 |
| 2026-08-14 onward | 18,517 | 0 | — | — | — | 18,517 |
| total | 21,191 | 1,642 | 1,572 | 23 | 47 | 19,549 |
All 1,640 pre-2026 gaps were requested without success. Of the 3,610 failed
targets, only 1,642 remain gaps under the no-artifact definition: 1,967 hold
PostScript or HTML (1,809 HTML), and `1808.02949` v1 holds nine PDFs under
names such as `v1.1.pdf`, although its standard version URL failed.
#### Recheck on 2026-09-13
The recheck covered 144 distinct version URLs, requested at fifteen-second
intervals. The rows below are disjoint; two URLs that appeared in two strata
were counted once and returned the same response on both requests.
| selected group | URLs | PDF | 404 | empty 200 | 500 |
|---|---|---|---|---|---|
| gaps dated 1991–1995 | 20 | 0 | 20 | 0 | 0 |
| all gaps dated 1996–2000 | 14 | 0 | 14 | 0 | 0 |
| gaps dated 2001–2010 | 20 | 0 | 15 | 4 | 1 |
| gaps dated 2011–2020 | 19 | 0 | 14 | 0 | 5 |
| gaps dated 2021 through 2026-08-13 | 20 | 18 | 0 | 1 | 1 |
| gaps dated 2026-08-14 onward | 20 | 20 | 0 | 0 | 0 |
| original empty-body cases | 12 | 0 | 0 | 12 | 0 |
| original HTTP 500 cases | 11 | 0 | 0 | 0 | 11 |
| held-version controls | 8 | 8 | 0 | 0 | 0 |
All 98 previously requested URLs repeated their earlier failure: 63 returned
404, 18 returned 500 and 17 returned an empty 200. All 38 previously unasked
URLs and all 8 controls returned PDFs, with no transport errors or throttling.
The failures persisted between the two checks. Untested URLs and later
availability remain unverified.
The 19,549 gaps never requested by the build are all dated 2026 and account
for 92.2% of the gap set. This query selects them together with two 2026
versions that returned HTTP 500:
```sql
SELECT paper_id, version, version_date, arxiv_pdf_url
FROM 'hf://datasets/secemp9/arxiv-complete/versions/*.parquet'
WHERE NOT is_metadata_only
AND NOT has_pdf AND NOT has_ps AND NOT has_html
AND version_date >= DATE '2026-01-01';
-- 19,551 rows
```
At fifteen seconds per request, a pass takes about 81 hours before retries.
If all 19,549 unasked versions yielded PDFs, retrievable-version PDF coverage
would rise to about 99.93%. The sample supports attempting these fetches, but
does not establish that outcome. Check arXiv's current access rules before
starting a new pass.
### Papers with no available text route
An inventory check found 13,912 papers (0.442%) with no published text route:
13,898 have no files and 14 have files but no usable text route. The other
3,134,884 (99.558%) meet at least one route below. The table counts matching
records without assessing their text quality: a `paper_text` row can still
be a stub, and a PDF can be blank or defective.
| cumulative route | papers added | still without a route |
|---|---|---|
| `paper_text` row | 2,856,227 | 292,569 |
| PDF in `pdf` | 278,276 | 14,293 |
| PostScript in `ps` | 15 | 14,278 |
| PDF-named file in `source` | 5 | 14,273 |
| PostScript-named file in `source` | 20 | 14,253 |
| HTML-named file in `source` | 340 | 13,913 |
| extensionless TeX document in `source` | 1 | 13,912 |
The first six routes can be reproduced from the indexes:
```sql
WITH reach AS (
SELECT paper_id,
max(kind = 'pdf') AS in_pdf,
max(kind = 'ps') AS in_ps,
max(kind = 'source' AND lower(path) LIKE '%.pdf') AS src_pdf,
max(kind = 'source' AND regexp_matches(lower(path),
'\.(ps|eps|epsi|epsf|pstex)$')) AS src_ps,
max(kind = 'source' AND regexp_matches(lower(path),
'\.(html?|xhtml|shtml)$')) AS src_html
FROM 'hf://datasets/secemp9/arxiv-complete/files/*.parquet'
GROUP BY paper_id
)
SELECT m.paper_id
FROM 'hf://datasets/secemp9/arxiv-complete/metadata/*.parquet' m
LEFT JOIN reach r USING (paper_id)
LEFT JOIN (
SELECT paper_id
FROM 'hf://datasets/secemp9/arxiv-complete/paper_text/*.parquet'
) t ON t.paper_id = m.paper_id
WHERE t.paper_id IS NULL
AND NOT coalesce(r.in_pdf OR r.in_ps OR r.src_pdf OR r.src_ps
OR r.src_html, false);
-- 13,913 rows, before the extensionless-document check
```
Adding `latex` or `.dvi` files as routes changes this result by zero. Indexed
`html` rows are excluded because their content config was omitted. Among the
2,864 papers with source but no PDF or PostScript row, 2,485 have
`paper_text`; the remaining 379 hold 6,596 source files (118,117,724 bytes).
All those files were read, classified from their bytes and checked against
their recorded lengths and hashes, with zero mismatches. Of the 379 papers,
363 ship HTML in `source`. Filename errors did not falsely add a text route
for any of this residual set.
The 2,864 source-only papers hold 10,930 source rows and 450,068,220 raw
bytes, with a median of one file. Their 3,624 version rows include 1,781 with
HTML and 1,457 non-zero-size gaps. They largely predate the snapshot edge:
1,427 first appeared in 1991–1995 and the newest on 2026-07-27. Their leading
categories are `hep-ph` (473), `hep-th` (369) and `cs.LO` (222).
The seventh route is `gr-qc/9408005`: its entire paper, *The Black Hole History
in Tamed Vacuum*, is 65,618 bytes of Phyzzx TeX in the extensionless file
`BHTAME`, opening `%%%%Phyzzx file` and closing `\end`. A later census found
extensionless TeX documents in 1,663 papers without `paper_text`; 1,659
already have PDFs and three have PostScript, leaving only this paper to add
to the table. See [extensionless files](#extensionless-files).
All 13,898 fileless papers first appeared in 2026, earliest on March 18;
12,965 appeared on August 14 or later. Their 14,788 version rows carry no
artifact and only 16 are metadata-only. This is principally snapshot lag.
Nine have `withdraw` in their comments, versus 8,713 papers corpus-wide, so
the evidence does not support describing the whole group as withdrawals.
The fourteen remaining papers are:
| paper | indexed files | reason no text route is available |
|---|---|---|
| `1804.09174` | `removed.txt`, 178 B | admin removal note |
| `1807.09532` | `removed.txt`, 154 B | admin removal note |
| `1810.01471` | `removed.txt`, 139 B | admin removal note |
| `1912.04194` | `removed.txt`, 162 B | admin removal note |
| `2409.04654` | `removed.txt`, 198 B | admin removal note |
| `2501.05533` | `removed.txt`, 198 B | admin removal note |
| `2512.09237` | `removed.txt`, 198 B | admin removal note |
| `nucl-ex/0701073` | `auto-ignore.txt`, 118 B | redirected to `nucl-th/0610004` |
| `q-bio/0606043` | `auto-ignore.txt`, 83 B | redirected to `q-bio/0602024` |
| `hep-th/9502028` | `9502028`, 231 B, plus 2 PNGs | resubmitted as `hep-ph/9502316` |
| `gr-qc/9410009` | `9410009`, 372 B, plus 2 PNGs | replacement notice |
| `1810.03044` | `1810.03044.bin`, 0 B | admin removal, empty payload |
| `1612.00844` | 10 PNGs, 2,306,178 B | no text file in the submission |
| `2608.02072` | one HTML file, 6,400 B | held by the build, omitted from publication |
All 31 files in the fifteen-paper residual set before the `BHTAME` correction
were read and re-hashed, with zero size or digest mismatches. Twelve of the
fourteen have withdrawal or removal remnants. Ten of the eleven notes start
with `%auto-ignore`; `gr-qc/9410009` instead starts `%Paper: gr-qc/9410009`.
The three redirected works are available under their replacement identifiers,
with 56,518, 37,298 and 30,500 characters of `paper_text`, respectively.
The other two cases are the figure-only submission and the unpublished HTML
proceedings. This count records which papers lack a published text route.
On 2026-09-14, the newest PDF URL for each of the fifteen candidates, plus
v1 for the three with multiple versions, returned 18 HTTP 404s. Four controls
returned PDFs. The check does not establish that no copy exists elsewhere.
This availability count exceeds the 3,133,383 papers with an indexed arXiv
render: 1,502 additional papers are reachable through their submission files,
while `2608.02072` has an indexed render whose bytes are not published.
Thus 3,133,383 + 1,502 − 1 = 3,134,884.
## Reading data
All SQL examples use DuckDB.
### Grouping files by paper
`latex`, `source`, `pdf` and `ps` have one row per file. Rows are sorted by
`paper_id`, contiguous for each paper and never split across shards. The
audit checked ordering and boundaries across all 295,391 row groups in the
four configs' 3,686 shards.
| config | rows | papers | mean rows/paper | median | p90 | p99 | p99.9 | max | paper at max |
|---|---|---|---|---|---|---|---|---|---|
| `source` | 47,373,604 | 3,121,338 | 15.18 | 8 | 33 | 118 | 371 | 16,440 | `1309.2485` |
| `latex` | 12,348,082 | 2,860,286 | 4.32 | 2 | 10 | 30 | 66 | 3,378 | `2303.11935` |
| `ps` | 2,237,839 | 1,318,462 | 1.70 | 1 | 3 | 8 | 25 | 1,533 | `0912.3460` |
| `pdf` | 4,973,956 | 3,131,959 | 1.59 | 1 | 3 | 5 | 8 | 187 | `0903.5132` |
Each maximum belongs to one paper. In `source`, 697,611 papers (22.35%) have
one row, 42,535 exceed a hundred and 358 exceed a thousand. The last group
contains 638,026 rows and 14.70 GB of raw content. Single-row shares are
38.20% in `latex`, 60.81% in `pdf` and 63.19% in `ps`.
To assemble each paper's LaTeX files:
```python
import itertools
from datasets import load_dataset
ds = load_dataset(
"secemp9/arxiv-complete", "latex", split="train", streaming=True
)
# Each contiguous paper_id group contains the paper's LaTeX files.
for pid, rows in itertools.groupby(ds, key=lambda r: r["paper_id"]):
files = {r["path"]: r["content"] for r in rows}
print(pid, len(files), sum(map(len, files.values())))
```
This loop runs to exhaustion. Read the
[early-exit caveat](#streaming-early-exit)
before shortening it with `break`.
The dictionary holds an entire paper's raw content. File count alone does not
predict its memory cost:
| config | median raw bytes/paper | p90 | p99 | p99.9 | largest paper, raw bytes | paper and rows |
|---|---|---|---|---|---|---|
| `source` | 752 KB | 7.69 MB | 37.6 MB | 54.2 MB | 3,325,641,586 | `2412.11029`, 32 |
| `pdf` | 921 KB | 7.46 MB | 39.5 MB | 104 MB | 2,291,310,505 | `2206.04656`, 7 |
| `ps` | 713 KB | 4.38 MB | 20.8 MB | 64.7 MB | 1,138,179,620 | `2505.01271`, 1 |
| `latex` | 99.4 KB | 346 KB | 1.07 MB | 44.1 MB | 138,236,581 | `2310.17514`, 149 |
With `pyarrow` 25.0.1, assembling local rows into that dictionary produced:
| source paper | rows | raw content | row groups used / shard total | peak process RSS |
|---|---|---|---|---|
| `1309.2485` | 16,440 | 175.1 MB | 83 / 277 | 366 MiB |
| `1606.00972` | 15,720 | 61.3 MB | 80 / 292 | 244 MiB |
| `2412.11029` | 32 | 3,325.6 MB | 4 / 63 | 6,141 MiB |
The bare-interpreter baseline was 66 MiB. `1309.2485` is the Model Checking
Contest 2013 report; 16,213 rows are PDF-named files totalling 158.2 MB, with
a median size of 7,550 bytes. Its row groups require 67.5 MB compressed and
195.6 MB uncompressed. The 32-row coupled-cluster submission `2412.11029`
has 28 ancillary files; its four groups require 158.9 MB compressed and
3,399.3 MB uncompressed.
In `source` and `latex`, grouping collects files from one unversioned package.
In `pdf`, it mostly collects revisions: 4,973,956 rows represent 4,973,947
version pairs. Only two pairs have multiple rows (`1808.02949` v1 has nine;
`1611.09018` v3 has a PDF and a text file). The 187-row maximum is therefore
187 revisions, and mean rows and versions per paper both round to 1.5881.
`ps` combines revisions and assets: 2,237,839 rows represent 1,953,094 pairs,
of which 1,882,090 have one row and 71,004 have more. That is 284,745 rows
above one per pair, with means of 1.481 versions and 1.697 rows per paper.
`0912.3460` has two versions, with 768 and 765 objects. The distinction from
its 293,617 asset count is explained under [PostScript](#postscript-content).
### Row groups and memory
The readers measured for this release load substantially more than one row
when reading Parquet content. Compressed row-group size estimates transfer
cost; uncompressed size is a useful starting point for memory budgeting.
Peak RSS can be several times larger because of decoding, prefetching and
conversion to Python objects. Small `batch_size` values did not remove that
cost in the tested readers.
These measurements cover all 301,750 row groups across all 3,745 shards.
Sizes are decimal MB; `rows/RG` is the packer's cap and commonest value.
Groups can be shorter at shard boundaries or for unusually large payloads.
| config | rows/RG | unc. median | unc. p90 | unc. p99 | unc. max | comp. median | comp. p90 | comp. max | over 100 MB unc. |
|---|---|---|---|---|---|---|---|---|---|
| `pdf` | 200 | 224.3 | 768.4 | 794.6 | 1,113.3 | 178.5 | 654.9 | 957.2 | 74.55% |
| `ps` | 200 | 203.9 | 371.9 | 565.6 | 1,383.6 | 48.1 | 100.8 | 413.5 | 72.50% |
| `latex` | 2,000 | 88.1 | 235.5 | 484.3 | 811.8 | 21.7 | 38.7 | 141.0 | 46.77% |
| `source` | 200 | 33.4 | 73.2 | 124.6 | 1,598.1 | 21.6 | 55.4 | 571.5 | 3.00% |
| `paper_text` | 500 | 44.2 | 55.1 | 61.3 | 100.3 | 12.4 | 14.9 | 29.1 | 0.02% |
| `metadata` | 50,000 | 71.4 | 79.9 | 82.9 | 83.1 | 26.9 | 30.3 | 32.2 | 0% |
| `files` | 200,000 | 22.2 | 23.3 | 26.7 | 28.2 | 8.8 | 9.4 | 10.5 | 0% |
| `versions` | 100,000 | 18.2 | 18.5 | 19.2 | 19.5 | 5.4 | 5.4 | 5.6 | 0% |
| `sample` | 200 | 18.1 | 21.4 | 21.7 | 21.8 | 5.0 | 5.8 | 5.9 | 0% |
Repacking the retained large groups would require rewriting 9.49 TB of
`pdf`, `ps` and `latex` Parquet, estimated at 20–24 hours plus upload.
In `pdf`, 38.23% of groups exceed 250 MB compressed. PostScript compresses
well but still has a 203.9 MB median uncompressed group.
With `datasets` 5.0.1 and `pyarrow` 25.0.1, fetching one row measured:
| read | transfer | peak RSS |
|---|---|---|
| first PDF from Hub, 1.3 MB content | 126 MB | 717 MiB |
| same, projecting out `content` | 11.6 MB | 181 MiB |
| first row of local PDF shard 00696, 0.43 MB content | — | 3,905 MiB |
| same local shard, projecting out `content` | — | 160 MiB |
The remote transfer comprised 11.6 MB of setup traffic and 114.4 MB of
content from two prefetched groups (56.6 MB and 57.7 MB). Shard 00696's
first group is 977.4 MB uncompressed over 187 rows; the result is not a
measured worst case for the whole config.
Project out `content` when you only need an index. Non-content columns
total 241 MB compressed for `pdf`, 108 MB for `ps`, 551 MB for `latex` and
2.36 GB for `source`. In `paper_text`, columns other than `text` total
1.24 GB. For example, this reads only metadata columns from a PDF shard and
can stop after one batch:
```python
from huggingface_hub import HfFileSystem
import pyarrow.parquet as pq
fs = HfFileSystem()
path = "datasets/secemp9/arxiv-complete/pdf/train-00000-of-01886.parquet"
with fs.open(path, "rb") as handle:
pf = pq.ParquetFile(handle)
batches = pf.iter_batches(
batch_size=64, columns=["paper_id", "path", "sha256"]
)
print(next(batches).to_pydict())
```
On a tested local PDF shard, this projection read 9.4 kB and peaked at
66 MiB RSS against a 60 MiB baseline; including content read 516 MB and
peaked at 1,155 MiB. `datasets` also accepts
`columns=["paper_id", "version", "path", "size", "sha256"]`.
By comparison, changing `datasets` to `batch_size=1` reduced one measurement
from 1,993 to 1,836 MiB on a 623.8 MB group. Arrow's
`iter_batches(batch_size=1)` still read 516 MB and peaked at 1,155 MiB in
the other test. A mid-file read of one 0.78 MB PDF required 609.5 MB of
reads and 2,007 MiB RSS from a 706.8 MB group.
Allow roughly 2–4 GB per PDF or PostScript reader for ordinary groups, with
more for the exceptional rows below. For selected papers, locate and query
their shards instead of streaming the whole config. For file categories
such as `anc/`, shard selection is much less effective; use a reader that
can evaluate a small column before fetching content.
### Streaming early exit
The release audit found interpreter-shutdown hangs or aborts when a
`datasets` iterator was abandoned. The tested environment was `datasets`
5.0.1, `pyarrow` 25.0.1 and Python 3.12.3. Results after taking one row over
the Hub and ending the script were:
| config | outcome | runs |
|---|---|---|
| `sample` | clean exit | 3 of 3 |
| `metadata`, `latex`, `pdf`, `ps` | `SIGABRT` | 3 of 3 each |
| `versions`, `paper_text`, `source` | hang | 3 of 3 each |
| `files` | hang | 12 of 12 |
The failure occurs after the row is delivered. It can leave a job running or
return a failed exit status; it is not evidence that the delivered row was
corrupted. The abort included `PyGILState_Release` during finalization. A
local 288 MB `files` shard also hung in 4 of 4 tests, excluding the network
as a necessary cause. An all-config process aborted in 11 of 11 tests.
An abandoned Parquet generator was observed finalizing after `pyarrow` had
been torn down, raising an `ArrowInvalid` attribute error. One hung process
had thirteen threads in `futex_wait_queue`, after 3 seconds of useful CPU
work and 282 seconds without further CPU progress.
Related upstream reports include [Arrow #45214](https://github.com/apache/arrow/issues/45214),
closed as a duplicate of [#49942](https://github.com/apache/arrow/issues/49942),
with fix [#49943](https://github.com/apache/arrow/pull/49943). Those reports
do not override the release's measurements: the audit reproduced failures on
25.0.1 and also tested 24.0.0. Results should be rechecked when changing
library versions.
| tested approach | outcome |
|---|---|
| `pq.ParquetFile(...).iter_batches(...)`, stopped early | clean, 4 of 4 |
| DuckDB, local or HTTP | clean, 3 of 3 |
| exhausting the iterator | clean, 4 of 4 |
| `it.close()` or `contextlib.closing(iter(ds))` | hang, 4 of 4 |
| deleting iterator/dataset then `gc.collect()` | abort, 6 of 6 |
| projecting with `columns=` | abort, 4 of 4 |
| pinning `pyarrow==24.0.0` | hang, 4 of 4; abort, 3 of 3 |
| direct `os._exit(0)` | exit, 6 of 6 |
| `atexit.register(lambda: os._exit(0))` | exit, 4 of 4 |
For partial reads, use DuckDB or the direct Arrow example above. Forcing the
older-version cleanup branch in `datasets` still aborted in 6 of 6 tests.
`os._exit` bypassed shutdown, but also bypasses buffer flushing and cleanup;
an unconditional zero-status exit can hide application errors. It is not a
general replacement for normal program termination.
### Largest rows
A reader returning `content` as one Python `bytes` value must fit that value
in memory, regardless of batch size. Repacking can reduce neighbouring data
in a row group but cannot reduce the size of the file itself.
The table covers all file sizes in `files` and UTF-8 encoded string sizes in
`paper_text`. Units are decimal.
| config | rows | median | p90 | p99 | p99.9 | max | >64 MB | >256 MB |
|---|---|---|---|---|---|---|---|---|
| `source` | 47,373,604 | 35.9 KB | 380 KB | 2.76 MB | 10.5 MB | 1,568.6 MB | 358 | 19 |
| `pdf` | 4,973,956 | 637 KB | 4.90 MB | 24.4 MB | 48.5 MB | 1,113.3 MB | 1,342 | 69 |
| `ps` | 2,237,839 | 428 KB | 2.52 MB | 12.3 MB | 40.9 MB | 1,138.2 MB | 466 | 27 |
| `latex` | 12,348,082 | 16.7 KB | 86.3 KB | 282 KB | 1.76 MB | 72.5 MB | 1 | 0 |
| `paper_text` | 2,856,227 | 68.5 KB | 156 KB | 367 KB | 977 KB | 45.9 MB | 0 | 0 |
Using 64 MiB (67,108,864 bytes) instead changes the counts to 338 in
`source`, 1,177 in `pdf`, 422 in `ps` and one in `latex`.
All six rows exceeding 1 GB were read back, hashed and inspected. Five exceed
1 GiB:
| raw bytes | config | file |
|---|---|---|
| 1,568,567,496 | `source` | `2412.11029`, `anc/CCSDTQ_os_12_r123t123_eqns_permu-R_pre.txt` |
| 1,141,065,026 | `source` | `2103.14690`, `anc/NHS_catalog_final.cds` |
| 1,138,179,620 | `ps` | `2505.01271v1.ps` |
| 1,113,320,441 | `pdf` | `2206.04656v2.pdf` |
| 1,113,320,091 | `pdf` | `2206.04656v1.pdf` |
| 1,040,178,656 | `ps` | `2303.11154v1.ps` |
The first two are ancillary text data. The rendered documents have valid
format headers and end markers. `latex`'s largest file is `2508.04390`'s
72,454,841-byte `anthology.bib`; `paper_text`'s largest row is `2109.06332`,
45,888,100 UTF-8 bytes and 45,888,090 characters.
None of the 1,937 indexed files above 64 MiB matches the hash of a zero-filled
buffer of its own size. The largest [NUL-filled file](#nul-filled-files) is
33,496,382 bytes, below this range.
The largest PDF sits alone in row group 9 of
`pdf/train-00740-of-01886.parquet`: 1,113,321,202 bytes uncompressed,
13,365,541 compressed. On `pyarrow` 25.0.1:
| operation | peak process RSS |
|---|---|
| read the Arrow row group with `content` | 2,208 MiB |
| also convert the value to Python `bytes` | 4,332 MiB |
| project out `content` | 70 MiB |
The Python result cost about four times the row's 1,061.8 MiB. The PDF's
image streams compress well in Parquet (83:1 for this group), so its small
network transfer does not imply a small memory requirement. The largest row
accounts for 99.99993% of its group in `pdf`, 99.4% in `ps` and 98.2% in
`source`. In `latex`, the 72.5 MB bibliography shares a 231.3 MB group with
1,999 other files.
Use the index to identify large values before fetching them:
```sql
SELECT kind, paper_id, version, path, size
FROM 'hf://datasets/secemp9/arxiv-complete/files/*.parquet'
WHERE size > 64000000
ORDER BY size DESC;
-- 2,166 rows, 256.9 GB of raw content
```
The six files above 1 GB occupy five shards: `pdf/train-00740`,
`ps/train-00100`, `ps/train-00115`, `source/train-00482` and
`source/train-01155`. Filtering after a streaming row arrives does not avoid
its read cost. If exact snapshot bytes are unnecessary, `arxiv_pdf_url` may
provide a smaller current rendering; `2206.04656v2` was 21,742,481 bytes
there on 2026-09-13. See [PDF rendering changes](#pdf-rendering-changes).
### Latest held versions
`pdf` and `ps` retain the revisions the build obtained. Superseded PDF rows
account for 3,798,892,308,546 of 10,371,932,999,324 raw content bytes
(36.6%) and 1,841,997 of 4,973,956 rows (37.03%). Those shares do not
measure compressed Parquet savings, because row groups mix revisions.
The 268,733,475-byte `versions` config supplies three flags:
| flag | meaning | true rows |
|---|---|---|
| `is_latest_version` | newest version in the harvested arXiv record | 3,148,796 |
| `is_latest_pdf` | newest version with a PDF held | 3,131,959 |
| `is_latest_ps` | newest version with PostScript held | 1,318,462 |
For 20,652 papers, the newest held PDF is older than the newest version in
the metadata: 0.659% of the 3,131,959 PDF-bearing papers, or 0.656% of all
papers. This comparison is within the snapshot; it is not a live check of
arXiv's current revision. Filtering only on `is_latest_version` would lose
these papers from a latest-held-PDF view.
```python
from datasets import load_dataset
v = load_dataset("secemp9/arxiv-complete", "versions", split="train")
latest = {(r["paper_id"], r["version"]) for r in v if r["is_latest_pdf"]}
pdf = load_dataset(
"secemp9/arxiv-complete", "pdf", split="train", streaming=True
)
# Membership selects the latest held version; all files of that pair are kept.
for row in pdf:
if (row["paper_id"], row["version"]) in latest:
print(row["paper_id"], row["version"], row["path"], row["size"])
```
The filter runs after transfer, so streaming still reads every revision.
Project out `content` if you only need the inventory, then retrieve selected
shards. The flags live in `versions` to avoid rewriting the 9.33 TB of PDF
and PostScript Parquet. For a partial run, use the alternatives described in
[streaming early exit](#streaming-early-exit).
## `paper_text`
`paper_text` provides one assembled TeX string for each of 2,856,227 papers.
It retains preambles, comments and macros; it is not plain prose extracted
from rendered pages. The 991-row `sample` has the same schema.
The remaining 292,569 metadata papers divide into 265,111 with source but
no `.tex` or `.ltx`, 13,560 with no source package, and 13,898 with no files.
The first group includes PDF-only submissions, encrypted `.cry` files and
1,663 papers with [extensionless TeX](#extensionless-files). Accounting for
those documents leaves 290,906 papers outside both `paper_text` and this
additional TeX route; the three set differences above are unchanged.
### Text size and token estimate
| measurement over all rows | total |
|---|---|
| Unicode characters (`len(text)`) | 246,780,951,228 |
| UTF-8 encoded bytes | 247,014,274,151 |
| bytes minus characters | 233,322,923, or 0.0945% |
The totals come from all 50 shards. There are 174,595,098 non-ASCII
characters (0.0707%); the remainder use one UTF-8 byte each. Encoded byte
length is distinct from Python object memory and from the 70 GB compressed
Parquet download. Raw UTF-8 content compresses by about 3.5:1.
The estimate of 78–81 billion tokens uses `cl100k_base`, a sample of
2,586 papers and observed ratios of 3.05–3.14 characters per token. It is
not a full token count. Using the measured character total gives 78.6–80.9
billion; other tokenizers need their own ratio:
```text
estimated tokens = 246,780,951,228 / measured characters per token
```
The original estimate used byte counts with a characters-per-token ratio.
This added 0.0945%, below the range's precision. A twelve-core benchmark of
`encode_ordinary_batch(num_threads=12)` in `tiktoken` processed 41.0 million
characters per second, suggesting about 1.7 hours for tokenization alone,
plus input time. A full pass was not run.
| statistic | characters per paper |
|---|---|
| minimum | 12 |
| p25 | 41,730 |
| median | 68,419 |
| mean | 86,401 |
| p75 | 105,915 |
| p90 | 155,575 |
| p99 | 366,153 |
| p99.9 | 974,568 |
| maximum | 45,888,090 (`2109.06332`) |
Quantiles are discrete order statistics. There are 2,694 papers over a
million characters, 800,271 at or above a hundred thousand, and 16,581 below
a thousand.
```python
from datasets import load_dataset
ds = load_dataset(
"secemp9/arxiv-complete", "paper_text", split="train", streaming=True
)
# This visits the full config; length is measured in Unicode characters.
for paper in ds:
print(paper["paper_id"], paper["title"], len(paper["text"]))
```
Row groups are substantially smaller than in the binary configs, but the
[early-exit issue](#streaming-early-exit) also affected `paper_text` on the
tested library versions.
### Assembly and omitted files
The builder follows `\input` and `\include` depth-first, in command order,
from a selected main file. It uses path-order concatenation when it cannot
find a main file or the expansion covers less than half the TeX bytes.
| `resolution` | meaning | rows | share |
|---|---|---|---|
| `single` | one `.tex` or `.ltx` file | 2,302,601 | 80.62% |
| `resolved` | main file selected and includes expanded | 520,797 | 18.23% |
| `fallback` | all TeX files concatenated in path order | 32,829 | 1.15% |
Unused files are excluded from resolved rows. They can be drafts, alternate
sections or the actual paper when the main-file selection is wrong.
`resolution` records the assembly method. Even a resolved row may omit the
paper or include unrelated text.
Bibliography databases (`.bib`) are excluded; formatted `.bbl` files are
appended if the include graph did not already incorporate them.
The builder leaves 465,530 TeX files out of 185,982 papers: 6.65% of its
6,998,107 input TeX files. Only `resolved` rows have non-zero
`n_unused_files`; 35.71% of those rows omit at least one. Among affected
papers, the median omitted count is 1, p75 is 2, p90 is 5 and p99 is 18.
The maximum is 1,323 for `2306.09110`, which uses 1,361 of its 2,684 files.
For every row, `n_tex_files = n_files_used + n_unused_files`. Totals are
6,998,107 = 6,532,577 + 465,530, matching `latex`'s 6,997,048 `tex` files
plus 1,059 `ltx` files. The 1,341,591 `.bbl` rows are outside these counts.
Every skipped input remains available in `latex`:
```sql
SELECT paper_id, path, size, sha256, content
FROM 'hf://datasets/secemp9/arxiv-complete/latex/*.parquet'
WHERE paper_id = '2306.09110' AND ext IN ('tex', 'ltx');
-- 2,684 rows
```
`main_file` names the selected entry point, but the used-file list is not
published. Recovering the exact omitted set requires resolving the include
graph again from the individual files.
### Repeated text and incorrect main-file selection
There are 2,844,004 distinct `text_sha256` values. Only 320 texts repeat,
across 12,543 rows. The largest group is the 11,369 identical bare
`%auto-ignore` stubs. Reading the other 319 groups found:
| repeated text, excluding the bare stub | groups | rows | characters |
|---|---|---|---|
| text carrying `%auto-ignore` | 110 | 443 | 13–2,107 |
| `\includepdf` wrappers | 90 | 403 | 121–1,580 |
| shared documents of at least 10,000 characters | 108 | 299 | 13,332–271,804 |
| other short text | 11 | 29 | 37–6,498 |
| total | 319 | 1,174 | |
Of the 108 long-text groups, twelve have at least three copies, accounting
for 107 rows; all twelve are publisher templates or macro packages. The
clearest case is the 77,099-character AIP `aipguide.tex`, selected as
`main_file` in 44 `resolved` rows. All 44 packages also contain the real
paper among three to five TeX files, such as `1411.4279`'s `paper.tex`,
`1106.5937`'s `heavy11.tex` and `1009.1634`'s `ms.tex`. Those papers remain
available in `latex` even though `paper_text.text` contains the guide.
Other repeated guides include IOP (14, 5 and 3 copies), MNRAS (7), World
Scientific (7), `jpconf` (7 and 5), Springer `EDITOR.TEX` (5),
`ws-procs975x65` (4), AMS-TeX (3) and IAU (3). The 96 two-copy groups
include `latex.ltx`, `mtexsis.tex`, `eplain.tex`, `amstex.tex`, `lineno.sty`
and ACM, LLNCS, AASTeX, CVPR, ICCV, NeurIPS, AISTATS and MICCAI templates.
Two matching texts alone are not proof of a resolver error.
The largest non-stub group by row count is a 134-character `\includepdf`
wrapper shared by 45 papers, from `1608.07343` through `1807.08563`.
Duplicate hashes are useful review candidates, but removing all duplicates
does not establish that the remaining rows contain the intended papers.
### Short rows and document markers
Of the 16,581 rows under 1,000 characters (0.58%), 13,235 contain
`%auto-ignore` and 3,078 contain `\includepdf`; these two sets are disjoint.
The remaining 268 were not exhaustively classified. Samples contain unmarked
withdrawal notes and empty preambles. There are 14,449 rows under 200
characters and 11,431 at 20 characters or fewer.
The wrappers have a median length of 192 characters. In 3,076 cases the
embedded PDF is present in `source`, totalling 5,733,788,446 raw bytes.
Their paper text must be extracted from that PDF. Filtering at
`length(text) >= 1000` removes this short-row population. Longer rows still
require quality checks.
An earlier revision of this page reported 2,677 wrappers, 2,675 with PDFs,
4.95 GB and 669 residual rows. Its intermediate file truncated text at 400
characters, hiding `\includepdf` in 401 rows. The counts above use the full
published strings.
A marker-only filter is unreliable. `%auto-ignore` appears somewhere in
14,751 rows, including 1,516 at least 1,000 characters long; 1,449 contain
it after the first character. Of the 13,302 rows that start with it, 154
exceed 10,000 characters and contain ordinary papers. The share of rows at
20 characters or fewer ranges from 0.02% to 0.94% across shards.
See [withdrawal and removal records](#withdrawal-and-removal-records) for
the complete set of recorded forms.
Document markers also need context:
- 182,539 rows (6.39%) lack `\documentclass`.
- 37,401 (1.31%) lack both `\documentclass` and `\begin{document}`.
- 150,623 of the first group use the older `\documentstyle`; 145,138
still contain `\begin{document}`.
- Counting both class-declaration spellings leaves 31,887 rows (1.12%)
with no declaration and no `\begin{document}`. Of these, 13,337 are
marker-bearing or short stubs, 7,757 are complete plain-TeX or AMS-TeX
documents, and 10,793 mix fragments, further plain TeX, HTML indexes and
non-text payloads.
The last share ranges from 0.29% to 9.92% across shards. These statistics
were measured over the full config; one shard is not representative.
### Decoding
The builder joins the selected files with newlines, then decodes the whole
paper once. It tries UTF-8 and falls back to latin-1 after any decoding
error. `text_encoding` records the branch used for the assembled paper.
Original file encodings are not recorded. One invalid byte can therefore
change the decoding of the whole document.
| branch | rows | share of rows | characters | share of characters |
|---|---|---|---|---|
| `utf-8` | 2,664,861 | 93.30% | 230,779,648,694 | 93.52% |
| `latin-1` | 191,366 | 6.70% | 16,001,302,534 | 6.48% |
The fallback rate ranges from 0.48% in shard 00044 to 17.92% in shard
00003. Its rows contain 83,056,146 non-ASCII characters, 0.519% of their
text but 47.6% of the corpus's non-ASCII characters. ASCII is unchanged by
either decoding branch.
Encoding the fallback strings back to latin-1 recovers the bytes presented
to the decoder. Across those bytes, 6,253,320 non-ASCII bytes (7.53%) belong
to valid UTF-8 multibyte sequences; 167,125 rows (87.3%) contain no such
sequence. That rules out UTF-8 as a complete decoding of those rows, but
does not establish latin-1 as the correct original encoding.
The audit grouped all fallback rows with heuristic byte tests:
| byte pattern | rows | share of fallback rows |
|---|---|---|
| non-ASCII only in `A0`–`FF`, compatible with ISO-8859-1/Windows-1252 | 94,075 | 49.16% |
| bytes in `80`–`9F`, decoded as C1 controls by latin-1 | 55,774 | 29.15% |
| runs of at least four non-ASCII bytes, consistent with CJK encodings | 19,070 | 9.97% |
| Mac OS Roman punctuation patterns | 12,642 | 6.61% |
| predominantly valid UTF-8, with mojibake from fallback | 9,805 | 5.12% |
Overlapping tests were assigned in this order: predominantly UTF-8, four-byte
run, C1 byte, Mac OS Roman, remainder. Without that precedence, 69,668 rows
contain a C1 byte, 52,113 a Windows-1252 smart-punctuation byte, 21,506 a
four-byte run and 18,170 the Mac OS Roman pattern. These tests provide clues
about source encodings; they do not identify every original codepage.
Among the 167,125 rows with no valid UTF-8 multibyte sequence, 56,730 still
contain C1 bytes. Examples place byte `96` where Windows-1252 uses an en
dash: `0709.2497` has `pp. 141<96>--147`, and `0807.0432` has
`reaction<96>diffusion`. Samples of the CJK-shaped group contain Shift-JIS,
EUC-JP, EUC-KR and GBK, often in comments. The full group was not decoded
file by file.
In the 9,805 predominantly UTF-8 rows, 707,201 valid UTF-8 characters
became 1,520,231 mojibake characters: 0.118% of those papers' text and
0.00062% of the corpus. For example, `1306.5649` joins 18 TeX files into
722,877 bytes. One invalid `ED` byte at offset 378,295 causes fallback,
misdecoding 4,648 otherwise valid non-ASCII characters. Use `latex` to
decode the individual source files with a different policy.
### Replacement characters already in the source
The UTF-8 branch is a strict decode, yet 16,076 of its rows contain
633,175 U+FFFD replacement characters. The builder does not use replacement
decoding, and none of the 191,366 latin-1 rows contains U+FFFD. These
characters were already encoded as `EF BF BD` in the input files.
Source checks, including `1909.05796` and `1211.6716`, found the same bytes
in `latex`; reading that config cannot recover the replaced characters.
The distribution is concentrated: 6,157 rows have one, 5,150 have two or
three, and 3,082 have four to ten. Just 33 rows contain over a thousand
each and account for 486,098 occurrences (76.8%). Across the UTF-8 branch,
the total is 0.000274% of its characters. Runs of at least two occur in
3,271 rows, and runs of at least three in 1,287.
The sparse cases appear in accented names and punctuation. The heavy cases
contain binary material embedded in TeX: `0802.2024` has 111,696 replacements
in 375,686 characters from a PDF inside `PrionBimodalHAL.tex`; `1005.0686`
and `1811.06523` also contain PDF material, while `1705.10234` and
`1210.1903` contain JPEG material. Four papers (`1803.00172`, `1901.01315`,
`2010.11203`, `2101.10652`) share a 287,489-byte gzip payload named
`JHEP3.cls.tex`, each with 47,494 replacements. Long rows can therefore
still contain non-text data.
### Licence populations
`license`, `title`, `abstract` and `primary_category` are copied from
`metadata`. The licence values agree on every joined row, but the
populations differ:
| licence column | NULL rows | total rows | NULL share | unit |
|---|---|---|---|---|
| `metadata.license` | 452,715 | 3,148,796 | 14.38% | paper |
| `paper_text.license` | 424,782 | 2,856,227 | 14.87% | paper with assembled TeX |
| `versions.license` | 649,392 | 5,030,612 | 12.91% | reported version |
Of the 292,569 papers outside `paper_text`, 27,933 (9.55%) have no recorded
licence. Papers with a recorded licence average 1.625 versions versus 1.434
for those without, explaining the lower version-level NULL share.
The arXiv non-exclusive licence occurs in 1,734,444 `paper_text` rows
(60.72%), compared with 60.42% of metadata papers. Each version repeats the
paper-level value; a per-version licence history was not harvested.
## Source content
`source` contains every extracted submission file, including figures, TeX,
code, supplementary data and build artifacts. Files were not filtered for
relevance. Most raw bytes belong to files named as figures or images:
| category by extension | files | raw content | share of raw bytes |
|---|---|---|---|
| figures / images | 31,911,285 | 8.426 TB | 88.53% |
| LaTeX-related files | 12,348,082 | 0.816 TB | 8.57% |
| build artifacts | 325,319 | 0.112 TB | 1.18% |
| everything else | 1,810,700 | 0.081 TB | 0.85% |
| video / audio | 8,543 | 0.050 TB | 0.53% |
| data / code | 969,014 | 0.031 TB | 0.32% |
| nested archives | 661 | 0.002 TB | 0.02% |
| total | 47,373,604 | 9.518 TB | 100.00% |
The exact raw total is 9,518,477,174,660 bytes. The compressed download is
6.51 TB. Categories use the lowercased final dotted alphanumeric component
of `path`, with disjoint extension sets:
| category | extensions |
|---|---|
| figures / images | `pdf` `png` `eps` `jpg` `jpeg` `gif` `ps` `svg` `tif` `tiff` `bmp` `pstex` `epsf` |
| LaTeX-related | `tex` `bib` `bbl` `sty` `cls` `bst` `clo` `cfg` `ltx` `tikz` `def` `ins` `dtx` |
| build artifacts | `synctex` `aux` `log` `out` `toc` `blg` `fls` `dvi` |
| video / audio | `mp4` `mpg` `mpeg` `avi` `mov` `mkv` `webm` `wmv` `flv` `m4v` `ogv` `mp3` `wav` |
| data / code | `py` `c` `cpp` `f` `m` `nb` `csv` `dat` `txt` `json` `sh` `java` `r` |
| nested archives | `rar` `xz` `7z` `tar` `gz` `zip` `zst` `tgz` `lzma` `bz2` `z` |
| everything else | remaining 7,122 extension strings, including the empty result |
The largest entries in the last category are `xml` (11.34 GB), no
alphanumeric extension (7.77 GB), then `ttf`, `fig`, `emf`, `ldf`, `pgf`
and `ai`. A suffix such as `.fdb_latexmk` falls in that category because it
contains an underscore.
```sql
SELECT count(*), sum(size)
FROM 'hf://datasets/secemp9/arxiv-complete/files/*.parquet'
WHERE kind = 'source'
AND lower(regexp_extract(path, '\.([A-Za-z0-9]+)$', 1))
IN ('pdf','png','eps','jpg','jpeg','gif','ps','svg','tif','tiff',
'bmp','pstex','epsf');
-- 31,911,285 files; 8,426,469,965,364 raw bytes
```
`kind = 'source'` and `source = 's3_src_extracted'` select the same rows.
These categories are based on filenames. Actual formats and uses can differ:
14,246,890 PDF-named files account for 3,614,260,349,571 raw bytes (38.0%
of `source`, about 43% of the figure category). Some are entire papers:
257,333 rows belong to packages without `.tex` or `.ltx`, totalling
428,146,685,889 bytes. See [PDF-only source](#pdf-only-source).
GIFs are assigned only to figures: 21,495 files, 5,375,567,265 bytes. In a
150-file random sample, 137 held one image, 11 were animated (12–500
frames), one could not be parsed to the end and one was an xv thumbnail.
Three animations were under `anc/`. Moving every GIF to video would change
the shares to 0.58% video/audio and 88.47% figures. An earlier table counted
GIFs in both categories, incorrectly inflating video/audio to 29,852 files;
the current categories are disjoint.
The video count uses the eleven container extensions above, excluding audio
and GIF: 8,000 files, 49,873,942,150 raw bytes, 2,646 papers. Of these,
5,796 are `.mp4` (29.94 GB). Restricting to `.mp4`, `.mpg`, `.mpeg`, `.avi`
and `.mov` gives 7,786 files and 48.44 GB. Audio adds 212 `.mp3` and 331
`.wav` files. Video formats have not been exhaustively identified; the
ancillary sample below found substantial misnaming among `.mpg`/`.mpeg`.
Other retained material includes 8,473 files in `.git/` directories across
58 papers (242,871,243 raw bytes), 3,185 Jupyter notebooks, and 1,141
`__pycache__/` files across 136 papers (7,871,857 bytes). Of the latter,
1,140 end in `.pyc`. Match `__pycache__` with `contains(path,
'__pycache__')`: SQL `LIKE '%__pycache__%'` also matches
`entropycache_parameter_ablation.tex`, producing an incorrect 1,142.
### Extensionless files
Under the alphanumeric-suffix definition, 273,784 files have no extension:
7,774,078,264 raw bytes, 93,544 papers and 201,422 distinct blobs. Of these,
120,421 have no dot (`README`, `Makefile`, `LICENSE`, `latexmkrc`); 153,363
have a non-alphanumeric final component, including 54,383 `.pdf_tex`,
25,117 `.pstex_t`, 20,275 `.fdb_latexmk` and 20,096 `.svn-base` files.
All 273,784 files were read from the published Parquet, covering 79,532
source row groups, and checked against their lengths and digests with zero
mismatches. Classification used signatures and TeX structure:
| content classification | files | raw bytes |
|---|---|---|
| plain text: READMEs, makefiles, licences, data, working copies | 182,224 | 4,585,979,570 |
| TeX fragments, including figure-export stubs | 61,007 | 306,047,196 |
| unrecognised binary | 9,654 | 581,093,462 |
| PostScript | 6,206 | 651,678,452 |
| PDF | 6,158 | 649,849,922 |
| complete TeX document | 4,599 | 215,705,737 |
| PNG, JPEG, TIFF, GIF or ICO | 3,219 | 572,051,846 |
| other formats, 154 partly textual files and 25 empty files | 717 | 211,672,079 |
| total | 273,784 | 7,774,078,264 |
Plain text and TeX fragments make up 243,231 files (88.8%); complete TeX
documents are 1.68%. The latter matter for `paper_text` coverage: among
2,174 extensionless files in 1,730 papers without a `paper_text` row,
1,675 files in 1,663 papers are complete documents, totalling 89,848,258
bytes. They comprise 1,612 LaTeX-shaped files and 63 plain-TeX files using
forms such as `\input amstex`, `\input harvmac`, `%%%%Phyzzx` or
`\magnification`, with `\bye` or `\end` endings.
The other 499 files comprise 232 texts without document structure, 194
PostScript or binary files, 67 under a kilobyte and six class/package files.
The 1,663 document-bearing papers date from 1994-02-18 to 2025-04-14;
761 predate 2000, with physics categories leading. Of them, 1,659 have a
PDF, three have PostScript and one has neither (`gr-qc/9408005`, the
`BHTAME` case). Another file brings 299 of them into `latex`, but none into
`paper_text`.
### Ancillary files under `anc/`
arXiv's [ancillary-file convention](https://info.arxiv.org/help/ancillary_files.html)
places supplementary data, code, images and other material in an `anc`
directory at the submission root. This release preserves that directory in
`source`. Most of its content is available in no other config.
Root-level `anc/` contains 149,573 files across 11,485 papers and
83,668,861,836 raw bytes: 0.32% of source files, 0.879% of source bytes and
0.37% of source-bearing papers. Select it with `starts_with(path, 'anc/')`.
`contains(path, 'anc/')` returns 150,597 because it also matches unrelated
names such as `MontBlanc/` and `BruitBlanc/` (172 files, 16 papers).
Another 852 files in 159 papers are under nested `/anc/` directories;
including them gives 150,425 files, 11,493 papers and 84,287,349,819 bytes.
The same disjoint extension sets used above give:
| category | files | raw bytes | share of ancillary bytes | share in all source |
|---|---|---|---|---|
| video / audio | 4,825 | 38,706,226,565 | 46.26% | 0.53% |
| data / code | 71,934 | 17,766,552,197 | 21.23% | 0.32% |
| figures / images | 19,013 | 17,221,419,442 | 20.58% | 88.53% |
| everything else | 50,094 | 8,963,221,645 | 10.71% | 0.85% |
| nested archives | 151 | 787,951,200 | 0.94% | 0.02% |
| build artifacts | 1,363 | 120,563,093 | 0.14% | 1.18% |
| LaTeX-related | 2,193 | 102,927,694 | 0.12% | 8.57% |
| total | 149,573 | 83,668,861,836 | 100.00% | |
This directory holds 77.18% of source's video/audio bytes, 57.85% of its
data/code bytes and 45.54% of its archive bytes. Its 4,790 video files
account for 38.64 GB. There are 2,003 extension spellings; leading ones are:
| extension | files | raw bytes |
|---|---|---|
| `mp4` | 3,408 | 21,357,092,270 |
| `pdf` | 11,983 | 13,759,090,070 |
| `mpg` | 357 | 10,087,990,225 |
| `txt` | 19,636 | 7,313,501,027 |
| `m` | 8,130 | 3,813,071,435 |
| `avi` | 503 | 3,435,195,665 |
| `csv` | 7,293 | 2,381,021,276 |
| `mov` | 354 | 2,238,164,512 |
| no extension | 4,288 | 2,050,436,422 |
| `gif` | 523 | 1,967,810,921 |
| `nb` | 1,862 | 1,921,514,257 |
| `dat` | 14,364 | 1,740,687,662 |
| `png` | 3,964 | 1,146,445,957 |
| `cds` | 2 | 1,141,073,122 |
| `html` | 1,806 | 431,551,936 |
| `json` | 10,920 | 389,433,494 |
| `tex` | 1,932 | 83,402,405 |
| `py` | 5,804 | 67,506,119 |
| `input` | 5,483 | 1,514,410 |
The PDF-named files span 4,918 papers; 3,097 have `supp` in their path
(2,844 papers), and 359 are named `si.pdf`. Common basenames include
`supplementary.pdf` (259 papers) and `supplement.pdf` (241). There are
1,607 README files across 1,167 papers.
A stratified sample of 2,111 ancillary files across 1,237 row groups was
read and hashed. Of 1,785 with extensions having a canonical format, 26
(1.46%) disagreed with their names. The other 326 used ambiguous names such
as `.dat`, `.db`, `.sav`, `.fig`, `.mx` or no extension. This sample covers
different formats from the general extension sample and is not directly
comparable to its 0.21% mismatch rate.
Sixteen of 61 sampled `.mpg`/`.mpeg` files were other containers (26%):
twelve ISO-BMFF and four AVI. For example, `1010.3081`'s
`anc/video1.mpg` is MP4, and `1310.0925`'s
`anc/TheShockAndTheTurbulence_HighResolution.mpg` is AVI. A sampled `.avi`
from `2511.23380` is ASF/WMV. Of 31 `.mat` files, 23 were MATLAB level-5,
one level-4 and seven plain text. One `.npy` was a 40-byte PAX record.
The remaining tested canonical formats matched, including 76 PDFs, 46 PNGs,
46 MP4s, 32 JPGs, 31 GIFs and 31 EPS files; 30 each of 7z, xz, FITS, HDF5,
ROOT, XLSX, DOCX, PS and JPEG; 22 WMV, 20 WebM, 19 MP3, 16 WAV and 14 RAR.
Signature checks need format-specific handling: six sampled tar files used
v7 headers without `ustar`, and three QuickTime files began with `mdat`
rather than `ftyp`.
Ancillary files by submission year and examples
Years below refer to the paper's first version. Its `anc/` files may have
arrived in a later revision. The 2026 row stops at August 27 (225,912
metadata papers), so it is a partial year.
| first submitted | papers with ancillary files | files | raw bytes | share of that year's papers |
|---|---|---|---|---|
| 2001 | 1 | 7 | 34,362 | 0.003% |
| 2007 | 1 | 42 | 163,309 | 0.002% |
| 2008 | 6 | 20 | 255,631,426 | 0.010% |
| 2009 | 9 | 373 | 108,385,874 | 0.014% |
| 2010 | 126 | 488 | 5,872,107,307 | 0.179% |
| 2011 | 194 | 1,109 | 4,996,001,989 | 0.253% |
| 2012 | 238 | 1,837 | 6,031,402,882 | 0.282% |
| 2013 | 297 | 2,403 | 5,777,034,123 | 0.320% |
| 2014 | 258 | 4,433 | 1,057,429,650 | 0.264% |
| 2015 | 315 | 11,150 | 978,023,374 | 0.300% |
| 2016 | 468 | 13,021 | 1,099,617,101 | 0.413% |
| 2017 | 656 | 4,850 | 1,492,020,821 | 0.530% |
| 2018 | 796 | 5,837 | 2,347,398,319 | 0.567% |
| 2019 | 831 | 9,092 | 3,001,434,810 | 0.533% |
| 2020 | 1,003 | 11,394 | 4,346,720,690 | 0.563% |
| 2021 | 980 | 9,062 | 8,476,760,223 | 0.540% |
| 2022 | 963 | 11,913 | 6,676,153,375 | 0.518% |
| 2023 | 945 | 13,904 | 7,005,635,311 | 0.452% |
| 2024 | 1,058 | 12,167 | 10,041,369,339 | 0.434% |
| 2025 | 1,170 | 14,153 | 8,375,229,823 | 0.412% |
| 2026 | 1,170 | 22,318 | 5,730,307,728 | 0.518% |
`0706.0489`'s 42 files arrived with its 2010 revision. The oldest example,
`cs/0106057`, has one version dated 2001-06-28 and seven Perl files for an
OAI-PMH harvester. Large file counts can come from a few submissions: two
papers contribute 5,561 and 3,560 of the 13,021 files in the 2016 row.
Examples read and re-hashed from published Parquet include:
- `2412.11029`, `anc/CCSDTQ_os_12_r123t123_eqns_permu-R_pre.txt`:
1,568,567,496 bytes of coupled-cluster coefficients; its 28 ancillary
files total 3.33 GB.
- `2103.14690`, `anc/NHS_catalog_final.cds`: 1,141,065,026 bytes of the
NEWFIRM HETDEX Survey catalogue in fixed-width CDS format.
- `2104.02716`, `anc/arxiv_programs/20_eqs_on_phi`: 680,955,946 bytes of
Mathematica equations.
- `2507.08247`, `anc/realizable-3terminal.txt`: 293,661,418 bytes of
combinatorial-game values for Hex positions.
- `1803.05205`, `anc/combinatorial3spheres9vertices_with_proofs.txt`:
194,667,981 bytes of enumerations and coordinate realizations.
- `2301.11363`, `anc/AmpSolid`: 219,324,297 bytes of symbolic amplitudes;
`2308.11686`, `anc/Spin_1_runGrid_45.dat`: 276,785,000 bytes of
Mathematica numerics.
- `2211.01987`, `anc/LaminatedK12-face-catalog.tar.xz`: 169,076,884 bytes;
7z certificates in `2507.01596`, `2602.17412` and `2010.11664` contain
49,486,732, 46,078,740 and 26,709,590 bytes respectively.
- `2607.11953`: 6,725 files under `anc/hidden-automata-rl/`, the largest
ancillary file count for one paper, with a reproduction README.
Fluid-dynamics papers account for 23,160,227,356 bytes (27.68%) across
473 papers, including 22,329,262,882 video bytes. By paper count the
leaders are `hep-ph` (1,026), `hep-th` (1,001), `cond-mat.mtrl-sci` (596),
`quant-ph` (585) and `cond-mat.mes-hall` (553).
The 149,573 files contain 137,167 distinct blobs and 388,123,271 duplicate
raw bytes (0.46%). Nine blobs also occur in `ps`: seven animated GIFs from
`1705.01852`, one JPG and one PNG. None occurs in `pdf`. The 2,193
TeX-related files also appear in `latex`; most ancillary content is exclusive
to `source`.
#### Fetching ancillary content
Inventory queries read only the small index columns:
```sql
SELECT paper_id, path, size, sha256
FROM 'hf://datasets/secemp9/arxiv-complete/files/*.parquet'
WHERE kind = 'source' AND starts_with(path, 'anc/');
-- 149,573 rows; 83,668,861,836 raw bytes indexed
```
Selecting whole shards saves little: 1,527 of 1,586 source shards hold an
ancillary row. Footer statistics also help little because paths are ordered
within each paper rather than globally: 232,644 of 243,083 groups have path
ranges overlapping `anc/`, allowing only 1.13% of compressed content bytes
to be excluded by that test.
The actual matching files occupy 11,681 groups, whose content chunks total
387,407,921,335 compressed bytes (5.95% of the content column). DuckDB can
read `path` first and avoid content chunks in groups with no matching row.
In a cold-cache test across eight shards spanning the corpus (00037, 00199,
00419, 00700, 00905, 01053, 01200, 01428), it read 1.71 of 31.48 GB
(5.43%), versus all 31.48 GB without the filter. Per-shard shares ranged
from 1.9% to 12.6%. On shard 00419, filtering on the affected paper IDs
instead read 1.33 GB versus 0.29 GB for the path predicate.
```sql
SELECT paper_id, path, size, content
FROM 'hf://datasets/secemp9/arxiv-complete/source/*.parquet'
WHERE starts_with(path, 'anc/');
```
Budget roughly 390 GB of Parquet reads for this approach. That delivers
83.67 GB of raw ancillary content along with the cost of neighbouring rows
in the same groups; the ratio is not a compression ratio. A Python filter
inside a `datasets` stream reads content before applying the filter and
therefore traverses the entire 6.51 TB. This distinction also matters for
[deduplication](#deduplication-and-transfer).
## LaTeX and duplication
### LaTeX composition
`latex` is exactly the subset of `source` with one of thirteen TeX-related
extensions. Both directions of the `(paper_id, path)` set comparison match,
as do the bytes and digests. This lets readers obtain TeX-related files in
160.23 GB of Parquet instead of downloading all 6.51 TB of `source`.
Bibliography databases account for more raw bytes than document source:
| `ext` | files | raw bytes | share | distinct blobs | papers | duplicate share of raw bytes |
|---|---|---|---|---|---|---|
| `bib` | 990,700 | 429,618,264,997 | 52.642% | 878,583 | 815,271 | 75.2% |
| `tex` | 6,997,048 | 226,676,162,698 | 27.775% | 6,675,745 | 2,855,728 | 2.2% |
| `cls` | 753,864 | 76,960,328,095 | 9.430% | 90,351 | 687,215 | 92.5% |
| `bst` | 840,854 | 33,209,017,599 | 4.069% | 30,802 | 636,124 | 95.0% |
| `bbl` | 1,341,591 | 25,417,106,079 | 3.114% | 1,299,902 | 1,288,037 | 2.3% |
| `sty` | 1,152,802 | 19,219,124,791 | 2.355% | 171,050 | 632,611 | 86.6% |
| `dtx` | 22,965 | 3,163,641,057 | 0.388% | 2,160 | 16,912 | 93.0% |
| `tikz` | 146,069 | 1,033,307,138 | 0.127% | 113,817 | 5,898 | 14.9% |
| `clo` | 65,321 | 424,789,970 | 0.052% | 1,630 | 46,284 | 95.5% |
| `def` | 12,272 | 292,972,380 | 0.036% | 2,155 | 4,293 | 89.5% |
| `ltx` | 1,059 | 33,868,730 | 0.004% | 909 | 705 | 4.0% |
| `cfg` | 6,663 | 32,965,623 | 0.004% | 1,869 | 3,883 | 73.3% |
| `ins` | 16,874 | 28,921,623 | 0.004% | 625 | 14,351 | 93.7% |
| total | 12,348,082 | 816,110,470,780 | 100.000% | 9,267,670 | 2,860,286 | 55.4% |
Distinct blobs and papers cannot be summed by extension: 1,828 blobs appear
under multiple extensions, so the per-extension blob counts total 9,269,598
rather than 9,267,670. Paper counts likewise overlap.
The `cls`, `sty`, `bst`, `clo`, `cfg`, `ins`, `dtx` and `def` files total
133,331,761,138 raw bytes, 92.3% of which repeat other files in the config.
The most repeated blob is `IEEEtran.cls` V1.8b: 281,957 bytes in 60,967
copies across 58,901 papers. Within the whole config, 75.06% of rows are
distinct blobs and 451,848,934,278 bytes (55.37%) are duplicates.
#### Bundled bibliography databases
Files selected by `ext = 'bib' AND contains(lower(path), 'anthology')`
account for 14,162 rows, 348,831,785,014 raw bytes, 2,816 distinct blobs
and 13,597 papers. They contain 81.2% of `.bib` bytes and 42.74% of all
`latex` bytes; one copy of each blob would total 34,906,441,967 bytes.
These are revisions and variants of the ACL Anthology bibliography bundled
with submissions.
The substring definition includes `acl_anthology.bib`, `aclanthology.bib`,
`filtered_anthology.bib` and `ACL_Anthology_part_aa.bib`. Restricting instead
to basenames beginning `anthology` and ending `.bib` gives 14,087 rows,
346,873,332,048 bytes, 2,781 blobs and 13,542 papers. The 75-row difference
uses 38 spellings, led by `acl_anthology.bib` (18) and `aclanthology.bib`
(13). No match arises solely from an `anthology` directory component.
There are 8,633 `.bib` files over 10 MB, totalling 358,326,271,199 raw
bytes, or 83.4% of `.bib`. They represent only 1,028 blobs; 88.8% of their
bytes are duplicate copies. Of these rows, 8,317 match the anthology
substring test (8,270 under the stricter prefix test). The other 316 rows
contain 9,804,072,995 bytes in 192 blobs, led by `crypto.bib` at 112 copies
and 3.57 GB.
One 43,619,593-byte `anthology.bib` occurs 2,095 times across 2,077 papers
and 42 path spellings within `latex`: 2,059 papers have one copy and 18 have
two. It contributes 91,339,427,742 duplicate bytes, 11.2% of the config's
raw bytes and 20.2% of its duplication. In `source`, the same blob has
2,099 copies across 2,081 papers and 46 spellings; the four additional
copies have `.bak`, `.txt` or extensionless names.
The twenty most copied `.bib` blobs are all anthology variants, together
contributing 230,829,650,496 duplicate bytes (51.1% of `latex` duplication).
All 1,028 over-10-MB blobs contribute 318,277,378,364 duplicate bytes:
98.5% of `.bib` duplication and 70.4% of the config's. The largest file is
the 72,454,841-byte anthology bibliography in `2508.04390`.
The large bibliographies occur in 8,441 papers (0.295% of `latex` papers).
Those papers' files total 360,539,676,092 raw bytes (44.2% of the config).
Removing just the 8,633 large `.bib` rows leaves 12,339,449 files and
457,784,199,581 raw bytes.
#### Filtered copies and download cost
The release audit re-encoded all 80 shards with zstd level 3, 2,000 rows
per group and dictionary encoding for `ext` and `paper_id`. The table gives
the measured sizes of those rewritten copies. Transfer costs for filtering
the existing shards are discussed below.
| retained rows | files | raw bytes | Parquet bytes | share of published size |
|---|---|---|---|---|
| all, published config | 12,348,082 | 816,110,470,780 | 160,231,046,567 | 100% |
| all except `.bib` | 11,357,382 | 386,492,205,783 | 93,069,995,195 | 58.1% |
| only `tex` | 6,997,048 | 226,676,162,698 | 63,724,684,922 | 39.8% |
| first occurrence of each digest | 9,267,670 | 364,261,536,502 | 94,280,445,323 | 58.8% |
A control re-encoding agreed with the published size within 0.014%; the
experimental copies were measured and deleted. Dropping all `.bib` saves
67,161,051,372 Parquet bytes (41.9%), whereas deduplication saves
65,950,601,244 (41.2%). Those are different filters. The large-bibliography
subset's raw-byte saving above does not establish its compressed saving.
The `tex`-only output is 63.7 GB, but a query cannot obtain it for that
transfer cost from the current layout. All 6,370 row groups contain multiple
extensions. Only one group's `ext` statistics exclude `tex`, allowing
5,899,008 of 159,671,084,826 compressed content bytes to be skipped
(0.0037%). Selecting `tex` therefore still reads nearly all content chunks.
If content is unnecessary, the other five columns total only 550,504,989
compressed bytes.
```sql
SELECT ext, count(*) AS n_files, sum(size) AS bytes,
count(DISTINCT sha256) AS blobs, count(DISTINCT paper_id) AS papers
FROM 'hf://datasets/secemp9/arxiv-complete/latex/*.parquet'
GROUP BY ext ORDER BY bytes DESC;
SELECT count(*) AS n_files, sum(size) AS bytes
FROM 'hf://datasets/secemp9/arxiv-complete/latex/*.parquet'
WHERE NOT (ext = 'bib' AND size > 10000000);
-- 12,339,449 files; 457,784,199,581 raw bytes
```
### Duplication by config
Every file occurrence is emitted, even when its bytes duplicate another
file. These figures group by `sha256` within each config and sum raw content
bytes before Parquet compression:
| config | rows | distinct blobs | distinct / rows | raw bytes | one copy per blob | duplicate bytes | duplicate share |
|---|---|---|---|---|---|---|---|
| `source` | 47,373,604 | 41,302,729 | 87.19% | 9,518,477,174,660 | 8,707,420,317,499 | 811,056,857,161 | 8.52% |
| `pdf` | 4,973,956 | 4,968,856 | 99.90% | 10,371,932,999,324 | 10,362,806,346,774 | 9,126,652,550 | 0.088% |
| `ps` | 2,237,839 | 2,202,253 | 98.41% | 2,680,486,270,056 | 2,677,023,865,270 | 3,462,404,786 | 0.129% |
| `latex` | 12,348,082 | 9,267,670 | 75.06% | 816,110,470,780 | 364,261,536,502 | 451,848,934,278 | 55.37% |
In `pdf`, 4,859 blobs repeat, none more than five times; 4,799 repeat only
within one paper. Byte duplication differs from superseded revisions:
36.6% of PDF raw bytes are older held revisions, while only 0.088% are
duplicate bytes. Use version flags for the former and digests for the latter.
Repeated `paper_text` strings have their own
[analysis](#repeated-text-and-incorrect-main-file-selection).
`latex` adds no files to `source`: its 451.85 GB of duplication is already
included in source's 811.06 GB. Downloading both configs adds 160.23 GB of
Parquet transfer to receive 816.11 GB of raw content again.
| corpus scope | emitted rows | raw bytes emitted | distinct blobs | one copy per blob, bytes | raw saving |
|---|---|---|---|---|---|
| `source` + `pdf` + `ps` | 54,585,399 | 22,570,896,444,040 | 48,013,629 | 21,294,542,858,112 | 1,276,353,585,928 (5.65%) |
| those three plus `latex` | 66,933,481 | 23,387,006,914,820 | 48,013,629 | 21,294,542,858,112 | 2,092,464,056,708 (8.95%) |
The `html` family is excluded because it has no content config; its inventory
contains 11,927 rows, 9,786 distinct blobs and 233,142,152 raw bytes.
`paper_text` contains assembled strings and is outside these file-level totals.
The per-kind blob counts sum to 48,473,838, exceeding the corpus total by
460,209 shared blobs: 252,421 occur in both `source` and `pdf`, 207,788 in
both `source` and `ps`, and none in both `pdf` and `ps`.
The `source`/`pdf` overlap contains 428,293,962,977 distinct raw bytes.
This differs from the 428,146,685,889 bytes of PDF-named source rows in
packages without TeX: they are different populations and aggregation units,
despite differing by only 147,277,088 bytes. They should not be substituted
or added.
On the source side of the PDF overlap, 252,468 of 252,495 occurrences are
PDF-named; the others are 25 `.bin`, one `.txt` and one extensionless file.
They span 252,291 papers, and 252,061 of 252,548 (blob, paper) pairings
match the same paper across both configs. The PostScript overlap largely
contains submission figures, with source-side occurrences named `.eps`
116,034 times, `.ps` 102,148, `.png` 70,429, `.jpg` 21,275, `.jpeg`
13,230 and `.gif` 12,241.
Similar overlap prevents adding extension-level dedup totals. The LaTeX
subset and its complement in `source` contain 451,848,934,278 and
358,786,357,231 duplicate bytes when deduplicated separately. Their sum is
421,565,652 below source's total because 4,953 blobs get a first copy in
both subsets. Deduplicate at the scope you intend to report.
#### Frequently repeated files
| copies across source/PDF/PS | bytes each | papers | content |
|---|---|---|---|
| 89,664 | 311 | 7,105 | OOXML slide-layout relationship file |
| 61,018 | 281,957 | 58,931 | `IEEEtran.cls` V1.8b |
| 53,297 | 213 | 53,297 | arXiv `00README.json` build specification |
| 44,996 | 20,521 | 43,570 | `fancyhdr.sty` |
| 30,095 | 45,154 | 29,420 | `natbib.sty` |
The 311-byte file points an unzipped PowerPoint slide layout to its master.
All 89,664 paths contain `_rels/` and `slideLayout`, under 20,486 distinct
spellings. Its copies occupy 27,885,504 bytes, of which 27,885,193 are
duplicates. `00README.json` records the top-level TeX file, TeX Live year
and compiler.
By duplicate bytes, the leading source blob is the 43,619,593-byte anthology
bibliography: 2,099 copies contribute 91,513,906,114 duplicate bytes
(11.28% of source duplication). Other revisions take the next three places.
Among source's 1,819,513 repeated blobs, the top 10 account for 28.48% of
duplicate bytes, the top 100 for 41.17% and the top 1,000 for 57.75%.
`IEEEtran.cls` contributes 17,204,170,269 duplicate bytes. The first
non-LaTeX entry is `IEEEtran_HOWTO.pdf`: 671,626 bytes in 8,120 copies,
or 5,452,931,494 duplicate bytes.
Scope explains the 61,018 corpus-wide versus 60,967 `latex` copies of
`IEEEtran.cls`: 51 copies across 50 papers have other names, including
`.bak`, `.old`, `.txt`, misspellings and eight extensionless `IEEEtran`
files. Within `ps`, the most repeated blob is the 286-byte `example.eps`, a
handwritten grey-square example dated 1997, with 2,488 copies across
2,035 papers. These representative blobs were read back and re-hashed.
### Deduplication and transfer
Plan deduplication against the 2.4 GB `files` index. This query also selects a
real representative `(paper_id, path)` pair for each digest:
```sql
SELECT sha256, any_value(size) AS bytes, count(*) AS copies,
min(paper_id) AS a_paper, arg_min(path, paper_id) AS a_path
FROM 'hf://datasets/secemp9/arxiv-complete/files/*.parquet'
WHERE kind = 'source'
GROUP BY sha256;
-- 41,302,729 blobs; 8,707,420,317,499 raw bytes
```
Independent `min(paper_id)` and `min(path)` expressions can select values
from different rows. `arg_min` keeps the path associated with the selected
minimum paper ID. A streaming pass can instead retain the first occurrence:
```python
from datasets import load_dataset
seen = set()
ds = load_dataset(
"secemp9/arxiv-complete", "source", split="train", streaming=True
)
# A digest is added once, yielding 41,302,729 distinct blobs on a full pass.
for row in ds:
if row["sha256"] in seen:
continue
seen.add(row["sha256"])
print(row["paper_id"], row["path"], row["sha256"])
```
This skips 6,070,875 output rows and 811,056,857,161 raw bytes after
transfer. Run to exhaustion or use a reader unaffected by the
[early-exit issue](#streaming-early-exit).
The full set of 41,302,729 SHA-256 strings peaked at 7,425 MiB process RSS
in the audit. Truncating to 128-bit strings measured 6,169 MiB and to 64-bit
integers 4,935 MiB, but truncated hashes weaken collision protection. The
example retains the full digest.
Stored `sha256` is a separate column: readers can evaluate it without reading
`content`. The limit here is layout. Under the first-occurrence strategy,
almost every row group contains a blob needed somewhere in the output:
| config | row groups | entirely duplicate groups | compressed bytes in those groups | shards with no new blob |
|---|---|---|---|---|
| `source` | 243,083 | 359 | 1,637,593,528 of 6,511,010,681,090 (0.025%) | 0 of 1,586 |
| `latex` | 6,370 | 0 | 0% | 0 of 80 |
| `pdf` | 33,047 | 0 | 0% | 0 of 1,886 |
| `ps` | 12,891 | 0 | 0% | 0 of 134 |
The most duplicate-heavy `latex` group is 98.3% duplicate but still contains
new content; no PDF group exceeds 25%. Whole-shard selection cannot exclude
any of the 3,686 shards under this strategy. This census does not prove an
optimal fetch plan for every possible choice of representative copies.
A Python `seen` filter saves output storage after the rows arrive.
The three primary file configs still transfer 15.84 TB of Parquet in a full
stream. A column-aware reader can avoid content chunks with no selected row,
as the [ancillary query](#fetching-ancillary-content) demonstrates.
Re-encoded `latex` retained 58.8% of the published Parquet size after removing
55.4% of raw bytes. The removed bytes had compressed at 6.85:1; retained
bytes at 3.86:1. `source` was not re-encoded, so its compressed deduplication
saving was not measured. The measured raw result for source/PDF/PS is
22.57 TB emitted versus 21.29 TB of distinct blobs.
## Schema
All configs use the `train` split. Content columns use Arrow `large_binary`
for file bytes and `large_string` for assembled text.
| config | columns |
|---|---|
| `metadata` | `paper_id`, `title`, `authors`, `abstract`, `categories`, `primary_category`, `submitter`, `license`, `doi`, `journal_ref`, `comments`, `report_no`, `msc_class`, `acm_class`, `proxy`, `n_versions`, `first_version_date`, `latest_version_date`, `oai_datestamp`, `oai_sets`, `arxiv_abs_url` |
| `versions` | `paper_id`, `version`, `version_date`, `version_date_raw`, `reported_size_kb`, `is_metadata_only`, `has_pdf`, `pdf_sha256`, `has_ps`, `n_ps_files`, `ps_bytes`, `has_html`, `n_html_files`, `source_version_proven`, `license`, `arxiv_pdf_url`, `is_latest_version`, `is_latest_pdf`, `is_latest_ps` |
| `files` | `paper_id`, `version`, `kind`, `source`, `path`, `path_encoding`, `size`, `sha256`, `stored_as`, `stored_sha256`, `stored_size` |
| `paper_text`, `sample` | `paper_id`, `text`, `main_file`, `resolution`, `n_tex_files`, `n_files_used`, `n_unused_files`, `text_encoding`, `text_sha256`, `title`, `abstract`, `primary_category`, `license` |
| `latex` | `paper_id`, `path`, `ext`, `size`, `sha256`, `content` |
| `source`, `pdf`, `ps` | `paper_id`, `version`, `kind`, `source`, `path`, `size`, `sha256`, `content` |
In `files`, `kind` identifies the artifact family and `source` its origin.
`path` is relative to the artifact's source layout.
`size` and `sha256` describe the published file bytes.
`stored_as`, `stored_size` and `stored_sha256` identify the original
gzip-wrapped mirror object where it differs; they are null otherwise.
These three columns exist only in `files`.
`version` is an integer and is null throughout `source`; `latex` has no
version column. S3 supplies one unlabelled source payload per paper.
Timestamp-based attribution was abandoned after matching only 6.4%.
`source_version_proven` instead records a source file matching a versioned
PDF byte for byte: it is true for 252,019 version rows (5.01%), all with
`has_pdf`, across 252,019 distinct papers. No paper has more than one
version marked by this method.
### Stored values
String filters are case-sensitive. In particular, `latex.ext` uses `tex`,
not `.tex`, and the two encoding columns spell UTF-8 differently.
| column | values or representation |
|---|---|
| `latex.ext` | `tex` `bib` `bbl` `sty` `cls` `bst` `clo` `cfg` `ltx` `tikz` `def` `ins` `dtx` |
| `files.kind` | `source` `pdf` `ps` `html`; content configs have a constant `kind` matching their name |
| `files.source` | `s3_src_extracted` `gcs` `web_scrape` |
| `source.source` | `s3_src_extracted` |
| `pdf.source` | `gcs` or `web_scrape` |
| `ps.source` | `gcs` |
| `files.path_encoding` | `utf8` or `percent` |
| `paper_text.text_encoding` | `utf-8` or `latin-1` |
| `paper_text.resolution` | `single` `resolved` `fallback` |
| `license` | one of nine full URL strings below, or SQL NULL |
| `primary_category` | 170 non-null values, such as `math.CO`, `cs.LG`, `hep-ph` |
| `metadata.categories` | space-separated string, such as `math.CO cs.CG` |
| `metadata.oai_sets` | non-empty list drawn from 158 OAI set specifications |
| `version` | integer from 1 to 187, except null source versions |
`ext` is the lowercased final dotted alphanumeric component of `path`, with
no leading dot, nulls, empty values or surrounding whitespace. All
12,348,082 rows agree with this extraction. Counts by extension are given
in the [LaTeX composition table](#latex-composition).
```sql
SELECT count(*)
FROM 'hf://datasets/secemp9/arxiv-complete/latex/*.parquet'
WHERE ext = 'tex';
-- 6,997,048; using '.tex' would return a count of 0
```
Of the 3,148,796 papers, 1,521,194 have multiple categories. Exact equality
with `cs.LG` selects 35,714 papers; testing membership in the space-separated
field selects 283,149. For this literal, `contains(categories, 'cs.LG')`
reproduces that count. When generalising to other strings, match complete
category tokens rather than arbitrary substrings.
```sql
SELECT count(*)
FROM 'hf://datasets/secemp9/arxiv-complete/metadata/*.parquet'
WHERE list_contains(string_split(categories, ' '), 'cs.LG');
-- 283,149
```
OAI set specs use a different grammar: `cs.LG` maps to `cs:cs:LG`, `hep-ph`
to `physics:hep-ph` and `cond-mat.mtrl-sci` to
`physics:cond-mat:mtrl-sci`. Of the 158 specs, 126 repeat the archive name,
20 nest a physics sub-archive and 12 have two components. Read the stored
list rather than constructing a spec from a category string.
Some source paths were not valid UTF-8, including CP949 directory names.
`path_encoding = 'percent'` marks their percent-encoded representation;
`urllib.parse.unquote_to_bytes` recovers the original path bytes. For literal
substring matching, `contains()` avoids the `_` and `%` wildcards in SQL
`LIKE`.
The nine recorded licence URLs, with metadata-paper counts, are:
| `license` | papers |
|---|---|
| `http://arxiv.org/licenses/nonexclusive-distrib/1.0/` | 1,902,375 |
| `http://creativecommons.org/licenses/by/4.0/` | 572,219 |
| NULL | 452,715 |
| `http://creativecommons.org/licenses/by-nc-nd/4.0/` | 88,143 |
| `http://creativecommons.org/licenses/by-nc-sa/4.0/` | 65,301 |
| `http://creativecommons.org/licenses/by-sa/4.0/` | 30,187 |
| `http://creativecommons.org/publicdomain/zero/1.0/` | 21,599 |
| `http://creativecommons.org/licenses/by/3.0/` | 7,912 |
| `http://creativecommons.org/licenses/by-nc-sa/3.0/` | 5,871 |
| `http://creativecommons.org/licenses/publicdomain/` | 2,474 |
`CC BY 4.0` is not a stored value. The same paper-level value is copied into
`versions`, `paper_text` and `sample`; see
[licence populations](#licence-populations)
for the different denominators and [LICENSE](LICENSE) for reuse terms.
### Field completeness
Counts below cover all metadata rows. *Populated* means non-null and
non-empty; it does not exclude sentinel strings such as `None`.
| metadata column | populated | share | NULL | empty string |
|---|---|---|---|---|
| `paper_id`, `title`, `authors`, `abstract`, `categories`, `primary_category`, `n_versions`, `first_version_date`, `latest_version_date`, `oai_datestamp`, `oai_sets`, `arxiv_abs_url` | 3,148,796 | 100% | 0 | 0 |
| `submitter` | 3,133,706 | 99.52% | 0 | 15,090 |
| `license` | 2,696,081 | 85.62% | 452,715 | 0 |
| `comments` | 2,267,796 | 72.02% | 881,000 | 0 |
| `doi` | 1,329,719 | 42.23% | 1,819,077 | 0 |
| `journal_ref` | 962,262 | 30.56% | 2,186,534 | 0 |
| `msc_class` | 397,255 | 12.62% | 2,751,541 | 0 |
| `report_no` | 192,813 | 6.12% | 2,955,983 | 0 |
| `acm_class` | 63,197 | 2.01% | 3,085,487 | 112 |
| `proxy` | 57,192 | 1.82% | 3,091,604 | 0 |
`oai_sets` is a list and is never empty. `submitter IS NOT NULL` retains all
rows, including 15,090 empty strings; `submitter <> ''` excludes them.
Of those empty submitters, 15,087 predate 2004. Live records for
`acc-phys/9411002` and `2001.10551` on 2026-09-14 also contained empty
submitter elements.
#### DOI and journal reference
DOI coverage varies by field and age. Filtering on `doi IS NOT NULL` selects
42.23% of the corpus and disproportionately retains older physics papers.
Rows below group primary categories by prefix and date by first submission:
| category prefix | 1991–1999 | 2000–2007 | 2008–2014 | 2015–2019 | 2020–2023 | 2024–2026 | all |
|---|---|---|---|---|---|---|---|
| `astro-ph` | 62.2% | 78.6% | 85.8% | 86.5% | 80.0% | 38.3% | 75.49% |
| `cond-mat` | 79.8% | 79.6% | 82.8% | 82.7% | 72.7% | 38.5% | 74.00% |
| `math` | 14.2% | 21.4% | 27.0% | 26.6% | 22.5% | 8.6% | 21.69% |
| `cs` | 6.1% | 13.4% | 22.8% | 19.4% | 16.7% | 9.5% | 14.53% |
Other overall rates include `hep-th` 73.90%, `gr-qc` 70.77%, `hep-ph`
69.61%, `quant-ph` 61.23%, `stat` 16.74% and `eess` 19.78%. The recent
decline is consistent with publication and metadata-update lag; these
percentages do not by themselves identify its cause or measure eventual
publication rates.
`doi` and `journal_ref` overlap but neither contains the other: 849,421
papers have both, 480,298 only a DOI, 112,841 only a journal reference and
1,706,236 neither. Their union is 1,442,560 papers (45.81%) with recorded
publication information. A paper's absence from this set does not establish
that it is unpublished.
#### Classification codes and sentinel values
`acm_class` has 63,309 non-null values, but 38,203 are the literal string
`None` and 112 are empty. The remaining 24,994 (0.79% of papers) are all
byte-identical to the corresponding `msc_class`. The combined fields add no
papers beyond the 397,244 with a code after excluding `None` and empty
values.
Live `arXivRaw` checks on 2026-09-14 reproduced this duplication for
`cs/0205064`, `math/0701684` and `2608.23263`; `q-bio/0312017` returned
`None` with no MSC element. Both columns preserve the
upstream values. Column names therefore do not reliably identify the coding
scheme: `cs/0205064` has ACM codes `F.2.2, F.1.1` in both, and
`math/0701684` has MSC codes in both. Of the 24,994, 24,190 begin with a
digit and 407 with a capital letter followed by a dot.
```sql
SELECT paper_id, msc_class, acm_class
FROM 'hf://datasets/secemp9/arxiv-complete/metadata/*.parquet'
WHERE acm_class NOT IN ('None', '');
-- 24,994 rows
```
After lowercasing and trimming surrounding whitespace, the sentinel values
`none`, `na`, `n/a` and `-` also occur in `comments` (191), `msc_class`
(89), `report_no` (15) and `journal_ref` (1). Adding `no`, `nil`, `null`,
`--` and `x` gives 237, 101, 16 and 1. These are too few to change the
printed completeness percentages, but applications should handle them.
Classification fields are also discipline-specific. Of the 397,244
code-bearing papers, 340,962 (85.8%) have `math` primary categories;
53.92% of 632,299 mathematics papers have a code. Computer science accounts
for 14,608 of the 24,994 non-sentinel `acm_class` rows (58.45%). Absence
of a classification code should not be treated as a uniform quality defect.
#### Proxy and version fields
`proxy` records a submission agent acting on an author's behalf. Its 57,192
rows contain 10,659 distinct values. The top five account for 37,823
(66.1%): `ccsd` 25,387, `vtex` 4,963, `EPTCS` 4,247, `auai` 1,899 and
`Sigma` 1,327. Spelling varies (`Sigma`, `sigma`, `Journal Sigma`), so
normalisation may be needed. A live check for `quant-ph/0402145` on
2026-09-14 reproduced `ccsd`.
Only two of `versions`' nineteen columns contain nulls:
| column | populated | share | NULL |
|---|---|---|---|
| other seventeen | 5,030,612 | 100% | 0 |
| `pdf_sha256` | 4,973,947 | 98.87% | 56,665 |
| `license` | 4,381,220 | 87.09% | 649,392 |
`pdf_sha256` is null exactly when `has_pdf` is false. None of the eight
boolean columns is null, and no string column contains an empty string.
Metadata completeness by first-submission date
| first posted | papers | licence | comments | DOI | journal ref | MSC | report no | ACM | proxy |
|---|---|---|---|---|---|---|---|---|---|
| 1991–1999 | 120,745 | 0.24% | 96.80% | 65.39% | 54.34% | 3.24% | 38.47% | 0.41% | 0.01% |
| 2000–2007 | 335,945 | 2.53% | 92.03% | 63.71% | 55.18% | 12.86% | 14.25% | 1.28% | 2.35% |
| 2008–2014 | 544,610 | 99.11% | 81.43% | 57.93% | 41.92% | 16.40% | 7.48% | 1.55% | 4.24% |
| 2015–2019 | 638,645 | 100% | 71.29% | 49.01% | 33.17% | 14.32% | 3.89% | 1.40% | 1.82% |
| 2020–2023 | 755,085 | 100% | 66.04% | 37.48% | 23.29% | 11.95% | 2.61% | 2.38% | 1.13% |
| 2024–2026 | 753,766 | 100% | 58.94% | 16.61% | 12.64% | 10.49% | 1.76% | 3.05% | 0.80% |
| all | 3,148,796 | 85.62% | 72.02% | 42.23% | 30.56% | 12.62% | 6.12% | 2.01% | 1.82% |
As in the main table, non-empty sentinel strings count as populated. After
excluding `None`, ACM rates become 0.00%, 0.00%, 0.53%, 0.63%, 1.09% and
1.31% respectively, for 24,994 papers overall.
The licence boundary is particularly sharp: all 2,682,895 papers first
posted on or after 2008-03-01 have a recorded licence. Of the 465,901
earlier papers, 452,715 (97.2%) do not; the latest first-submission date
among papers with a null licence is 2008-02-29. A licence-based selection
therefore changes the corpus's age distribution substantially.
```sql
SELECT count(*) AS n_rows,
count(doi) AS doi_not_null,
count(submitter) AS sub_not_null,
count(*) FILTER (WHERE submitter <> '') AS sub_not_empty,
count(acm_class) AS acm_not_null,
count(*) FILTER (WHERE acm_class NOT IN ('None','')) AS acm_real
FROM 'hf://datasets/secemp9/arxiv-complete/metadata/*.parquet';
-- 3,148,796; 1,329,719; 3,148,796; 3,133,706; 63,309; 24,994
```
## Provenance and verification
### Content handling
No missing PDF was generated by compiling TeX or converting another
representation. File contents were preserved, with three PDF replacements
and removal of gzip transport wrappers as described below. `paper_text` is
a derived, decoded string; its assembly is documented separately.
Three corrupt mirrored PDFs were replaced with complete copies from arXiv:
| paper | version | replacement note |
|---|---|---|
| `2401.18030` | 3 | mirror held a 267-byte HTML redirect under a PDF name |
| `0911.0496` | 1 | incomplete mirrored PDF replaced |
| `1001.4505` | 1 | incomplete mirrored PDF replaced |
The redirect pointed to `browse.arxiv.org/pdf/2401.18030v3`. Reconstructing
its recorded 267 bytes reproduces both the repair-journal SHA-256 prefix
`2e066757…` and the mirror-manifest MD5 prefix `db732060…`, establishing
what the replaced object contained. The published replacement is 321,907
bytes with a PDF header and end marker.
No published column marks these three replacements. Their `source`
remains `gcs`, and their paths retain the mirror layout. Published `size`,
`sha256` and `versions.pdf_sha256` describe the replacement bytes.
`repaired_from` existed only in an internal index; select these three
explicit `(paper_id, version)` pairs to identify them. The 6,475
gap-filling PDFs instead carry `source = 'web_scrape'`.
Gzip wrappers were removed from 2,168,515 objects, all `kind = 'ps'`:
2,070,188 named `.ps.gz`, 90,409 `.eps.gz` and 7,918 with other names.
The original name, size and digest remain in `files.stored_as`,
`stored_size` and `stored_sha256`.
Of those objects, 2,168,162 grew when decompressed, one stayed the same
length and 352 shrank. The shrinking objects are 3–154 bytes, across 27
papers: 41,149 bytes published versus 46,101 stored, with a maximum
reduction of 39 bytes. Gzip overhead can exceed its compression saving for
small inputs. All 62 objects published at 64 bytes or fewer shrank; 290 of
786 above 64 and below 256 bytes did; none of the 2,167,667 at 256 bytes
or more did. The equal-size object is `2201.10203` v1's 179-byte
`gyro_Yao_PRL_Revised.out.ps`. Consequently, `stored_size` need not be
less than `size`.
The build stored identical blobs once internally, but published configs emit
every occurrence. No submission file was removed solely because another
paper contained the same bytes.
### The 54 flagged PDF candidates
The build inspected the first kilobyte and last two kilobytes of all
19,227,355 objects named `.pdf` case-insensitively. It flagged 13,968:
13,035 without a header, 752 without an end marker in the tail, 179 with a
header at a non-zero offset and two empty files. Of these, 13,871 were
inside source packages. The remaining 97 rows represented 95 distinct
directly held objects; two hardlinked files were encountered twice.
Of the 95 objects, 28 had offset headers and 13 were PostScript figures
named `.pdf`. The other 54 candidates entered a comparison against arxiv.org
on 2026-09-05, at fifteen-second request intervals. Not all 54 were
defective PDFs:
| outcome | objects | publication |
|---|---|---|
| same defective bytes returned by arxiv.org | 42 | retained as received |
| complete copy obtained from arxiv.org | 3 | replacements listed above |
| intact PNG/JPEG figures named `.pdf` | 8 | retained in `ps` |
| partial temporary file included by the scan | 1 | never published |
| total | 54 | |
The 42 retained PDFs span 42 papers and 58,461,737 bytes. Each had a
`%PDF-` header but no `%%EOF` in its last two kilobytes, and each live
comparison returned an identical SHA-256. This establishes that arxiv.org
served the same bytes on that date; it does not establish that no other copy
exists. No schema column flags these files.
Two of them, `1304.1318` v1 and `1311.4268` v1, are `web_scrape` rows:
arxiv.org served those bytes on August 30 and again on September 5. The
observed defects therefore predate dataset packing.
The eight misnamed figures are five distinct blobs across four papers,
totalling 1,091,027 bytes:
| path suffix in `ps` | version(s) | bytes each | format |
|---|---|---|---|
| `1710.02099v1.PMerr_hist.pdf` | 1 | 18,258 | JPEG |
| `2204.12431v.blank.pdf` | 1, 2 | 4,440 | PNG |
| `2205.14881v.example.pdf` | 3, 4, 5 | 281,968 | PNG |
| `2308.02626v5.Figure1_flatsolution.pdf` | 5 | 105,052 | JPEG |
| `2308.02626v5.Figure2_deadcore.pdf` | 5 | 112,933 | JPEG |
The comparison script failed to extract an identifier from their `ps/`
paths, then requested `/pdf/None` eight times. Those 404s say nothing about
the figures' availability. All four papers have their rendered PDFs for
every version reported in the snapshot. These figures are eight of the
21 PDF-named objects in `ps`, none of which is actually PDF.
The temporary-file candidate was `tmp.1702.02106v3.pdf`, 18,190,224 bytes
without an end marker. Its path is absent from `files`. The published
`1702.02106v3.pdf` is the complete 20,092,270-byte mirror object
(`sha256 8b41aea2…`), with thirteen readable pages and a valid 126-entry
cross-reference table at offset 20,089,591. Its row was read directly from
`pdf/train-00205-of-01886.parquet` and matched the recorded digest.
arxiv.org returned a different complete 20,091,658-byte rendering
(`sha256 5207ce79…`) on September 5 and again September 14. Thus the
comparison's fourth “repairable” classification was a temporary-file false
positive. Of the 54 candidates, 45 were defective PDF objects: 42 retained
and three replaced. The other nine were not defective published PDFs.
## Verification
The build audit read, decompressed and re-hashed 69,789,708 content rows
across 3,736 shards, finding zero digest mismatches and zero unreadable
shards. Digests had been recorded during indexing, before packing.
| config | rows in the content hash pass | shards |
|---|---|---|
| `source` | 47,373,604 | 1,586 |
| `pdf` | 4,973,956 | 1,886 |
| `ps` | 2,237,839 | 134 |
| `latex` | 12,348,082 | 80 |
| `paper_text` | 2,856,227 | 50 |
| total | 69,789,708 | 3,736 |
The separate 991-row `sample` shard is outside that total. All 991 rows
match `paper_text` on paper ID and text digest, and their published strings
also pass direct UTF-8 digest verification. The three index configs contain
no file-content column.
The byte totals refer to different scopes:
| scope | raw content bytes |
|---|---|
| `source` + `pdf` + `ps` | 22,570,896,444,040 |
| the same plus indexed `html` | 22,571,129,586,192 |
| four file-level configs, including the repeated `latex` subset | 23,387,006,914,820 |
| those four plus UTF-8 encoded `paper_text` | 23,634,021,188,971 |
The last two round to 23.39 TB and 23.63 TB. Neither is the 16.08 TB
compressed download total.
For the four file-level content configs, structural checks reconciled row
counts, `(paper_id, path)` sets in both directions and every stored byte
length against the corresponding inventory. No missing or invented rows
were found. Shards were present, readable, sorted and non-overlapping, with
no duplicate keys.
For a file row, hash `content`. For an assembled text row, hash its UTF-8
encoding:
```python
import hashlib
def verify_file_row(row):
return (
len(row["content"]) == row["size"]
and hashlib.sha256(row["content"]).hexdigest() == row["sha256"]
)
def verify_text_row(row):
return (
hashlib.sha256(row["text"].encode("utf-8")).hexdigest()
== row["text_sha256"]
)
```
`text_sha256` validates the published string. Source files have separate
digests in `latex`. For the 191,366 latin-1 fallback rows, re-encoding as
UTF-8 changes the non-ASCII
bytes from those presented to the decoder. Use the individual `latex` rows
and their digests to verify original source bytes.
### Verifying your download
`SHA256SUMS` at the repository root lists every Parquet file. From a complete
download's root directory:
```bash
sha256sum -c SHA256SUMS
```
For a filtered download, with the manifest available in the same root:
```bash
sha256sum --ignore-missing -c SHA256SUMS
```
The latter checks only files that are present; it does not certify that the
requested subset is complete. File checksums verify transfer integrity,
while row-level digests verify the content inside Parquet.
## File quality and omissions
The sections below distinguish absent files, incomplete contents and unusual
but valid files. None of these conditions has a general-purpose quality flag
in the schema. The [coverage section](#snapshot-and-coverage) describes
fileless papers and version gaps;
[text quality](#short-rows-and-document-markers)
describes short rows and resolver limitations.
### PDF-only source
There are 265,111 source-bearing papers without `.tex` or `.ltx`, so they
do not appear in `paper_text`. Of these, 261,052 are also absent from
`latex`; the remaining 4,059 have other TeX-related files, including `.cls`
in 3,039 papers, `.sty` in 1,669 and `.bbl` in 908. These counts overlap.
Testing only for `.tex` gives 265,610 papers because 499 use `.ltx` instead.
Most of this group submits the paper as PDF: 251,258 (94.8%) have exactly
one source file and it is a PDF. In total, 252,815 have a PDF in source,
259,382 have PDF or PostScript, and 264,717 have an arXiv-rendered PDF in
`pdf`. PDF-named source files contribute 428,146,685,889 of the group's
439,623,556,691 raw bytes. These figures do not count images alone; source
PDFs can contain the full paper.
The group also includes other cases. In particular, 4,090 papers have
`.cry` files described below, and
[extensionless documents](#extensionless-files)
account for part of the TeX-name shortfall. Absence of recognised extensions
does not prove absence of TeX content.
`metadata` minus `latex` is a different set of 288,510 papers: it also
includes 13,898 papers without any files and 13,560 with no source package.
### `.cry` files
`source` contains 7,387 `.cry` files across 4,393 papers, totalling
358,179,037 raw bytes and 7,385 distinct blobs. Most names append `.cry`
to a TeX-related suffix: 6,675 `.tex.cry`, 187 `.pstex_t.cry`, 142
`.bbl.cry`, 119 `.sty.cry`, 69 `.cls.cry`, 26 `.bst.cry` and a longer
tail. Four have no preceding dotted component. None enters `latex` through
its `.cry` suffix.
A systematic sample of 247 files, one in thirty in `(paper_id, path)`
order, was read from published Parquet and re-hashed with zero size or
digest mismatches. Median entropy was 7.995 bits per byte (range
5.857–7.999), with 52–71% non-printable bytes. No sampled file had a
recognised signature or decompressed as gzip, zlib, bzip2 or xz.
Several comparisons support an encrypted TeX interpretation for these
TeX-named files:
- Three files have a same-named cleartext twin in the same package, two in
`1007.0328` and one in `0902.4344`. Each pair has equal lengths and
different digests.
- All 22 `boxedeps.tex.cry` files are 33,533 bytes, matching a clear
`boxedeps.tex` blob, but all 22 encrypted blobs differ. Sixteen
`ieeetran.cls.cry` files have the 201,353-byte length shared by 7,937
clear `IEEEtran.cls` files.
- The 122 `includepdf.tex.cry` files have only 24 distinct 97-byte prefixes.
Prefix groups correspond to neighbouring IDs posted within minutes, such
as `1205.6691`–`1205.6699` over 82 seconds and
`1203.6400`–`1203.6406` in under three seconds. Whole blobs also repeat
in neighbouring papers: `1005.1062`/`1005.1065` share an
`IEEEtran.cls.cry`, and `1211.1505`/`1211.1506` share `macros.tex.cry`.
For `1203.6400`–`1203.6406`, each package contains a PDF and a
143–150-byte encrypted wrapper. Reconstructing the wrapper template with
each package's PDF filename gives the matching length. XOR comparisons
recover the shared prefix stream and the other filenames, supporting the
interpretation of transformed LaTeX. The streams diverge after differing
input: `1203.6404` and `1203.6405` name
`p646_goetzgraefe_vldb2012.pdf` and `p656_goetzgraefe_vldb2012.pdf`,
differing at offset 99; their encrypted files agree through the first
99 bytes and differ thereafter.
The key source, transform and reason for these files' presence in arXiv's
archives were not established. No published key or general decoder is
available. The known-plaintext experiment does not recover arbitrary files.
Of the 4,393 papers, 4,090 lack plain `.tex` and `.ltx`; 4,088 of those
contain `.tex.cry`. All 4,393 have a rendered PDF in `pdf`, so this affects
source access rather than PDF availability. First versions run from 1995 to
January 2013, apart from `2602.15793`, whose ancillary `.cry` file may be
an unrelated use of the suffix.
### PostScript content
`ps` means objects held under the mirror's `/ps/` tree. It contains both
rendered PostScript and submission assets. After removing the
`v.` prefix from each basename, 1,944,222 of 2,237,839 rows
(86.9%) are the version's own `v.ps` render. The other 293,617
(13.1%) are assets across 60,609 papers, totalling 61.57 GB, or 2.30% of
the config's 2.68 TB raw content.
| asset names | counts |
|---|---|
| PostScript-related | `ps` 125,966; `eps` 90,413; `pstex` 1,032; `epsi` 344; `epsf` 207; `mps` 114; `eps_col` 104; `epsx` 52; `ps_col` 39; `ps~` 38; `eps2` 34; `cps` 31; `psc` 27; `ps_tex` 26 |
| other common names | `png` 28,742; `jpg` 22,992; `gif` 15,733; `jpeg` 1,659; `tex` 860; extensionless 679; `prn` 229; `xcp` 210; `pfb` 186; `bb` 91; `tif` 57; `enc` 35; `pro` 31; `ai` 24; `pdf` 21; `tikz` 19; `tiff` 12 |
The fourteen PostScript spellings total 218,427, with another 119 numbered
variants bringing that group to 218,546. The sixteen other spellings plus
extensionless files total 71,580, with another 3,491 objects across 1,214
spellings bringing that group to 75,071. Together the groups give 293,617.
Many tail suffixes are bare numbers from figure names.
Of the 1,953,094 indexed PostScript version pairs, 8,872 have assets but no
render. They contain 29,023 assets across 7,397 papers. This explains why
the 293,617 assets exceed the 284,745 rows above one per pair:
293,617 − 284,745 = 8,872. Subtracting one row per pair otherwise assumes
a render where none exists.
Byte checks covered 1,431 objects. All 400 sampled renders and 385 sampled
PostScript-named assets began `%!PS`. So did 452 of 646 assets sampled from
other names, including every sampled `.tex`, `.prn`, `.xcp`, `.pfb`, `.bb`,
`.enc` and `.pro`; small `.ai`, `.tikz`, `.plt`, `.md`, `.bib`, `.old`,
`.ep` and `.post` groups were read in full. All 21 files named `.pdf` were
also read: thirteen were PostScript, five PNG and three JPEG.
Raster-named assets total 69,195 files and 4.57 GB, 0.17% of raw config
bytes. Of 182 sampled, 179 matched the extension and three were PNGs named
`.jpg` or `.jpeg`. Extension counts are exhaustive; sampled signatures do
not establish the actual format of every untested file.
PostScript contributes little additional version coverage beyond PDF:
159 version pairs across 137 papers have `ps` but no PDF at that version.
Of those papers, 63 have a PDF at another version and 74 have none at any
version. Thus the approximately 99.99% overlap concerns version coverage.
The `pdf` and `ps` configs share zero byte-identical blobs.
### Filename and format mismatches
Extensions are useful selection fields but do not certify format. The build's
PDF-name census examined 19,227,355 objects and found 13,035 without a
`%PDF-` header in the first kilobyte. Of these, 13,013 are published source
files: 0.091% of the 14,246,890 PDF-named source rows, across 3,116 papers,
totalling 3,533,363,287 raw bytes. Every one matched a published row and size.
| actual content of PDF-named source files | files | raw bytes |
|---|---|---|
| PNG | 10,939 | 3,123,374,900 |
| JPEG | 954 | 298,418,775 |
| PAX extended-header records | 299 | 35,322 |
| PostScript | 195 | 44,569,236 |
| AppleDouble sidecars | 186 | 152,918 |
| starts with NUL bytes | 184 | 24,880,758 |
| text or source code | 177 | 23,023,963 |
| HTML, XML or SVG | 37 | 5,586,458 |
| other binary | 19 | 8,416,919 |
| git-lfs pointers | 17 | 2,216 |
| LaTeX with a `%PDFLaTeX` comment | 2 | 127,490 |
| BMP | 2 | 2,082,156 |
| MP4 | 1 | 1,958,936 |
| OLE compound document | 1 | 733,240 |
Of the 184 NUL-prefixed files, 109 are entirely NUL-filled:
16,509,598 bytes across 23 papers. They are a subset of the corpus-wide
[913 NUL-filled files](#nul-filled-files).
A published-Parquet recheck read 3,966 flagged source files from 400 shards
and 1,435 row groups, with zero length or digest mismatches. This was 28.6%
of the 13,871 source files flagged by all PDF checks (13,013 without a
header, 705 without a tail marker, 151 with offset headers and two empty).
The 13,013 format mismatches are clustered: 53% occur in papers holding at
least ten, and `2009.08576` alone contributes 334.
The other 22 no-header objects are the 21 misnamed figures in `ps` and the
267-byte redirect for `2401.18030` v3, which was replaced. The published
PDF tree contains one non-PDF, `1611.09018v3.txt`; all its `.pdf`-named
objects have PDF headers, though some remain defective.
A separate uniform sample of 900 source row groups covered 176,417 rows.
Of these, 164,663 used the 24 figure and TeX extensions below:
| name | sampled | mismatches | observed alternatives |
|---|---|---|---|
| `.jpeg` | 494 | 3.44% | 17 PNG |
| `.jpg` | 7,721 | 1.88% | 140 PNG, 2 PostScript, 2 xv thumbnails, 1 BMP |
| `.png` | 32,536 | 0.39% | 123 JPEG, 2 xv thumbnails, 1 PDF |
| `.eps` | 23,641 | 0.09% | 10 PJL streams, 7 PDF, 3 PNG, 2 LaTeX stubs |
| `.ps` | 3,935 | 0.05% | 1 PJL stream, 1 TeX file |
| `.tex` | 25,473 | 0.04% | 5 HTML, 4 high-entropy binary, 1 PAX record |
| `.pdf` | 50,828 | 0.03% | 16 PNG, 1 JPEG |
| other seventeen extensions | 20,035 | none observed | `.bbl`, `.sty`, `.bib`, `.bst`, `.cls`, `.tikz`, `.svg`, `.clo`, `.gif`, `.dtx`, `.ins`, `.cfg`, `.def`, `.tiff`, `.bmp`, `.ltx`, `.tif` |
The sample contained 339 mismatches (0.21%) among the selected extensions.
Other formats were outside its scope. Its PDF mismatch rate (0.033%) differs
from the full census (0.091%); use the census for that population. The JPEG
estimate rests on only seventeen mismatches.
A minimal signature helper for bytes already in memory is:
```python
MAGIC = (
(b"\x89PNG\r\n\x1a\n", "png"), (b"\xff\xd8\xff", "jpeg"),
(b"GIF87a", "gif"), (b"GIF89a", "gif"),
(b"%!PS", "ps"), (b"\xc5\xd0\xd3\xc6", "ps"),
(b"II*\x00", "tiff"), (b"MM\x00*", "tiff"),
(b"BM", "bmp"), (b"\x1f\x8b", "gzip"),
(b"PK\x03\x04", "zip"), (b"Rar!", "rar"),
(b"\x00\x05\x16\x07", "appledouble"),
)
def sniff(content):
if b"%PDF-" in content[:1024]:
return "pdf"
# Each listed prefix identifies a candidate format, not file validity.
for signature, kind in MAGIC:
if content.startswith(signature):
return kind
return None
```
This is not a validator or a complete format detector. It allows PDF headers
at a non-zero offset, unlike an offset-zero test. A bare `%!` is ambiguous:
`%!TEX root = ...` is a TeX editor directive, while `%!` alone can head
PostScript. EPS can also begin with a bounding box or operators. `P7 `
identifies xv thumbnails. A previous detector treating every `%!` as
PostScript inflated the sampled TeX mismatch rate to 1.17%; the corrected
rate is 0.04%. Case-fold filename suffixes: 4,257 of the 19,220,877
PDF-named paths in the published inventory use mixed or uppercase suffixes.
#### PDF end markers
The original scan flagged 752 rows with a header but no `%%EOF` in the last
two kilobytes: 750 distinct objects, because two hardlinked objects were
encountered twice. Of the flagged rows, 705 were inside source packages;
47 described directly held objects, of which 45 were distinct.
A recheck of that flag list classified 495 rows as truncated, with no
`%%EOF` anywhere. In the other 257, the marker was followed by data: 181
had padding, 49 binary data and 27 an appended HTML page. Many PDF readers
tolerate trailing data, so a missing tail marker alone is not proof of
truncation. Another 179 rows had an offset header (151 in source); an
offset-zero signature test would misclassify them.
Two PDF-named source rows were empty: `2309.07108`'s `stdin.pdf` and
`2505.19488`'s `images/transformer.pdf`.
Of the 752 flagged rows, 58 had exact power-of-two lengths: seventeen at
1 MiB, fifteen at 64 KiB, fourteen at 512 KiB, four at 256 KiB, three each
at 128 KiB and 32 KiB, one at 2 MiB and one at sixteen bytes
(`2403.07721`, `figures/fig-overview.pdf`). This pattern is consistent with
buffer-boundary truncation; it does not identify the responsible component.
Fifteen were rendered PDFs returned byte-identically by arxiv.org during
rechecks, 42 were source-package files and one was a repeated encounter of
a rendered object. The direct-object outcomes are accounted for in the
[54-candidate analysis](#the-54-flagged-pdf-candidates).
### PDF rendering changes
There are 1,342 `pdf` objects over 64,000,000 bytes, across 1,071 papers:
0.027% of rows but 150.7 GB, or 1.45% of PDF raw content. Of these, 250
exceed 128 MB, 69 exceed 256 MB and sixteen exceed 512 MB. The largest,
`2206.04656v2`, is 1,113 MB versus a config median of 0.64 MB.
On 2026-09-13, the audit compared sizes using `HEAD` requests to the exact
version URLs for the eight largest objects and 32 randomly selected from the
remaining 1,334. All eight largest were bigger in the snapshot, by
9.1–66.6 times. Among the random 32, 26 had matching reported sizes, one
differed by 218 bytes and five were 8.5–21.8 times larger here. A matching
size does not prove byte identity. The other 1,302 were not tested.
For the pair examined in detail, `2206.04656v2`, both files have the same
28 pages and 85 images at identical pixel dimensions. The live file was
21.7 MB, with a render date of 2023-02-11 versus 2022-06-13 here, and
compressed image streams stored essentially uncompressed in the snapshot.
That finding applies to this pair. `arxiv_pdf_url` provides the live route
when a current rendering is preferable to exact snapshot bytes.
### HTML omission
`files` indexes 11,927 `html` objects across 1,459 papers and 233,142,152
raw bytes, all from `gcs`, but there is no `html` content config. The family
was carried through indexing and omitted from the packer's config list;
the build records no deliberate exclusion decision. All 11,927 held
objects were re-read and matched their recorded sizes and digests during
the audit. They could be published in a later revision.
The family comprises pages and assets in 2,024 version directories. A full
signature census found 9,388 raster images (8,544 GIF, 601 JPEG, 242 PNG,
one WebP), 106 PostScript files, eleven PDFs, 2,407 markup files (2,364
documents and 43 fragments), four PCL streams and eleven other text/source
files. There are 2,406 HTML-named paths.
The 11,927 rows contain 9,786 distinct blobs and 27,287,667 duplicate
bytes (11.70%). For example, `quant-ph/9909090` v1 stores one 60-byte GIF
under sixteen names. Some bytes are available through source copies: the
four PCL streams in `patt-sol/9303002`, `patt-sol/9303004`,
`patt-sol/9303005` and `patt-sol/9304001` match files in `source`, totalling
7,525,635 bytes. Source contains eight `.pcl` files across six papers,
7,890,617 bytes in total.
Of the 1,459 HTML-bearing papers, 1,024 have `paper_text`, 109 have a PDF,
45 have PostScript, 382 ship HTML in source (518 files, 18,092,706 bytes),
52 ship a PDF in source and 38 ship PostScript in source. These sets
overlap. Only `2608.02072`, the LOCO 2026 workshop proceedings, has no
published text route: its sole held object is the 6,400-byte
`arxiv/arxiv/html/2608/2608.02072v1/loco2026-proceedings.html`.
`has_html` is true for 1,988 reported versions: 1,987 without PDF or
PostScript across 1,458 papers, plus `0802.0522` v1, which also has PDF.
`sum(n_html_files)` is 11,891. The remaining 36 inventory rows (740,031
bytes) belong to 36 version pairs absent from the OAI record and therefore
from `versions`. These flags describe the held inventory. The HTML content
config is absent.
### Withdrawal and removal records
Withdrawals and administrative removals appear in five recorded forms: a
bare marker, a marker followed by a note, a note under another filename,
a PDF notice, and an empty source payload. Filenames affect whether the
record reaches `paper_text`; the note's text identifies who removed the
submission and why. Because source payloads are unversioned, the held source
may be a replacement notice even when an earlier PDF remains available.
#### Bare markers
Hash comparisons found 11,497 files containing only `%auto-ignore` and
optional trailing whitespace. Of these, 11,416 contain exactly the twelve
bytes, across 11,415 papers, and 81 add whitespace. There are 11,479
`.tex`-named files and 11,423 named exactly `.tex`.
In `paper_text`, 11,430 rows contain this stub: 11,369 are exactly twelve
characters and 61 add whitespace. One further row adds a LaTeX `\\`, giving
11,431 rows at twenty characters or fewer. No row is shorter than twelve
characters or empty, blank or null.
For all 11,369 exact-stub papers, the held source package consists solely
of the twelve-byte `.tex`. Each has multiple versions and at least
one metadata-only version; for 10,884, that is the newest reported version.
`metadata.comments` contains `withdraw` for 5,215 of the 11,369 (45.87%),
compared with 8,713 of all 3,148,796 papers (0.277%) and 3,498 of the
3,137,427 non-stub papers (0.111%).
For the other 485, the newest version is not metadata-only; 477 have a PDF,
consistent with reposting after withdrawal and a stale held source stub.
Earlier PDFs remain available for 11,355 of the exact-stub papers: 16,703
rows and 23,533,143,198 raw bytes.
#### Marker followed by text
There are 13,302 rows beginning with `%auto-ignore`. Subtracting the 11,430
bare/whitespace stubs leaves 1,872 rows with further text. Of these, 1,317
are under 200 characters and another 362 under 1,000. Examples include
author withdrawals, publication problems and administrative removals, such
as `cs/0003063`, `astro-ph/0104484` and `gr-qc/0502058`.
The marker does not always mean withdrawal: 154 of the 13,302 rows exceed
10,000 characters and contain ordinary papers. Treat it as a candidate
signal rather than an unconditional exclusion rule.
#### Notes under other names
A filename audit, combined with inspection of single-file source packages
at most 8,192 bytes whose file was not named `.tex`, identified
97 papers whose submission is a removal or withdrawal note. Ninety-six have
only the note; `2603.12203` also has arXiv's 298-byte `00README.json`.
All notes were read from published Parquet and checked against their hashes
and lengths; all begin with `%auto-ignore`, four after leading whitespace.
| note filename | papers | note length |
|---|---|---|
| `removed.txt` | 79 | 115–218 bytes |
| `auto-ignore.txt` | 4 | 41–506 bytes |
| `.txt` | 4 | 44–151 bytes |
| `WithdrawnPaper.txt` | 2 | 85 or 86 bytes |
| `withdraw.txt` | 2 | 13 or 92 bytes |
| `removed.tex` | 2 | 199 or 219 bytes |
| `withdrawn.txt`, `ms.txt`, `evolvability.txt`, `version_vide.tex` | 1 each | 90–231 bytes |
Only three enter `paper_text`: `2305.02155`, `2308.13987` and
`math/0603115`, whose notes end in `.tex`. The other 94 do not, although
88 of the 97 papers have a rendered PDF. Additional notes in
`hep-th/9502028` and `gr-qc/9410009` accompany figures and fall outside the
small-package selection.
Files named `removed.tex` or `deleted.tex` in ordinary multi-file packages
can be author drafts rather than removal notices.
Eighty-three notes name arXiv administrators or carry an admin signature;
fourteen do not. The administrative reasons are:
- 66: submitter lacked the right to agree to the licence;
- 9: copyright infringement;
- 2: author-policy violations;
- one each: external text overlap, an unconsenting co-author, duplicate
submission, inaccurate authorship, names added without authorship, and
removal following IEEE Xplore publication.
All 79 `removed.txt`, both `removed.tex` and the one `withdrawn.txt` fall
in the administrative group. `auto-ignore.txt` splits one administrative to
three others. The remaining names belong to the unsigned/author group.
Examples include `physics/0401074` (temporary withdrawal at an advisor's
request), `cond-mat/0310326` (different results) and `0905.4570` (a mistake
in the previous version). `q-bio/0606043` does not identify who removed it.
Fourteen is the count of notes without an administrative signature.
#### PDF notices and empty payloads
The audit read and extracted text from all 2,290 source packages consisting
of one PDF at most 64 KiB. Sixteen yielded at most 400 characters. Eleven
contain only an explicit withdrawal notice, including `math/0310247`,
`quant-ph/0609176`, `cond-mat/0504111`, `physics/0501147` and
`quant-ph/0611120`; five print `%auto-ignore` on the page. A twelfth,
`cond-mat/0211524`, promises a later complete version.
The other four produced no extracted text: `1009.4523` has one blank page,
`quant-ph/0504128` declares zero pages, and `cs/0010015` and `2401.11525`
could not be opened by the tested `pypdf` reader. None of these sixteen has a
`paper_text` row, but all have a rendered PDF row.
The fifth recorded form is `1810.03044`: its whole source payload is the
zero-byte `1810.03044.bin`, with an administrative copyright-removal note
in metadata. It is one of the
[242 zero-byte source files](#zero-byte-and-small-files).
The first two forms account for 13,302 of 2,856,227 `paper_text` rows
(0.47%); 11,430 (0.40%) are bare or whitespace-only stubs. These figures
include the long marker-bearing rows noted above. The other forms require
looking beyond `paper_text`, and previous PDFs often remain available.
### NUL-filled files
There are 913 non-empty files consisting entirely of zero bytes, all in
`source`: 215,358,402 raw bytes across 189 papers and 719 distinct blobs.
They account for 0.0017% of indexed files, 0.00095% of indexed raw bytes
and 0.0060% of papers. Some are valid empty archives or application records;
zero-filled does not always mean damaged.
The census compared each recorded digest with the SHA-256 of a zero-filled
buffer of the same length. Computing these digests incrementally over the
3,533,564 distinct sizes requires hashing only as many zero bytes as the
largest size, 1,568,567,496. For example:
```python
import hashlib
def zero_hashes(sizes):
block = b"\x00" * (1024 * 1024)
digest = hashlib.sha256()
previous = 0
# Incremental updates must equal sha256 of n zero bytes at each size n.
for n in sorted(set(sizes)):
if n < 0:
raise ValueError("File sizes must be non-negative")
remaining = n - previous
# Chunking preserves that digest while bounding the temporary buffer.
while remaining:
take = min(remaining, len(block))
digest.update(block[:take])
remaining -= take
previous = n
yield n, digest.copy().hexdigest()
```
Match only `size > 0` for the NUL-filled population; size zero belongs to
the separate empty-file population below. The index census and independent
content-config column checks agree: `source` 913, `latex` 86, `pdf` zero,
`ps` zero. Every one of the 913 was then read from published Parquet across
178 shards and 268 row groups. All lengths and digests matched, and every
byte was zero. Distinct lengths and distinct blobs both equal 719.
| provenance | indexed files | raw bytes | NUL-filled files |
|---|---|---|---|
| `gcs` | 7,217,247 | 13,038,658,919,994 | 0 |
| `web_scrape` | 6,475 | 13,993,491,538 | 0 |
| `s3_src_extracted` | 47,373,604 | 9,518,477,174,660 | 913 |
All 913 were also located in the original unmodified source archives for
the 189 papers. Every original member was already entirely NUL-filled;
none differed or was missing. This rules out zeroing by this dataset's
extraction and packing steps.
The 86 files also in `latex` contain 36,188,700 bytes across 34 papers:
38 `tex`, 18 `bst`, 10 `sty`, eight `tikz`, seven `bib`, two `bbl` and one
each of `cls`, `ins`, `dtx`. `1802.09064` contains 29 of them. Neither
`paper_text` nor `sample` has an entirely NUL-filled string; their digests
were checked against zero-filled strings through length 46,000,000.
The audit classified the files as follows:
| classification | files | raw bytes | papers |
|---|---|---|---|
| valid empty tar archives | 71 | 229,376 | 69 |
| Krita default-pixel records | 42 | 162 | 5 |
| remaining files with missing content | 800 | 215,128,864 | 115 |
| total | 913 | 215,358,402 | 189 |
The empty archives include 54 files of 1,024 bytes and seventeen of
10,240 bytes, corresponding to tar end blocks and padding. GNU tar's
default empty archive is 10,240 zero bytes (`sha256 84ff9269…`). Of the
71, 52 have `.tar` or `.tgz` names and nineteen are extensionless. All 101
NUL-filled files whose length is a multiple of 512 opened as zero-member
archives with Python `tarfile`; that parser result alone does not establish
that every such file was intended as an archive.
The 42 Krita `layerN.defaultpixel` records belong to unpacked Krita
documents. Zero is the correct default for a transparent layer. All 42
files with this suffix in the corpus are NUL-filled. Among the remaining
files, the largest are `2107.14589`'s `anthology.bib` (33,496,382 bytes),
`1410.1323`'s `arXiv.tar` (27,140,749) and `2209.11629`'s `arxiv.tar`
(15,405,436).
This population does not show the large power-of-two pattern seen in flagged
PDFs. Although 139 files have power-of-two lengths, the largest is 32,768
bytes and 54 are the 1,024-byte empty archives. None is 64 KiB, 128 KiB,
256 KiB, 512 KiB or 1 MiB. The observed alignment is mainly tar's
512-byte blocks;
the cause of the other zero-filled content was not established.
All 189 papers have a rendered PDF, and 186 have `paper_text`. The median
affected paper has one NUL-filled file among 32 source files. Three papers
have only an empty source tar (`1607.05755`, `1702.06740`, `2008.05843`),
all the same 10,240-byte blob; two are marked withdrawn in comments. Their
PDFs are still present.
Entire directories are affected in three papers: 22 `(paper, directory)`
pairs containing multiple files are all NUL-filled. These are
`1802.09064`'s `samples/` (25 files, 2,121,825 bytes) and
`content/Archived/` (eleven), nineteen `imgs/…` directories in
`2406.05561`, and `2406.05558`'s `pictures/DACH` (five).
NUL-filled files by common filename suffix
| suffix | files | raw bytes | papers |
|---|---|---|---|
| `png` | 327 | 26,718,446 | 19 |
| `pdf` | 109 | 16,509,598 | 23 |
| `jpeg` | 69 | 29,496,155 | 1 |
| `tar` | 63 | 78,300,878 | 63 |
| no extension | 50 | 394,229 | 35 |
| `defaultpixel` | 42 | 162 | 5 |
| `tex` | 38 | 322,975 | 7 |
| `eps` | 32 | 14,071,809 | 9 |
| `bst` | 18 | 932,355 | 12 |
| `txt` | 15 | 13,283 | 3 |
| `jpg` | 13 | 4,191,068 | 6 |
| `sty` | 10 | 102,885 | 9 |
| `svg` | 9 | 465,949 | 2 |
| `bib` | 7 | 34,374,365 | 6 |
There are 62 suffix spellings in total, plus fifty extensionless files.
The 68 `.tar`/`.tgz` files and 109 PDF-named files discussed elsewhere are
subsets of these 913. Affected papers date from 1997-01-22 to 2026-07-30;
141 first appeared in 2018 or later. Leading categories are `cs.LG` (14),
`eess.SY` (8), then five categories at seven papers each.
### Zero-byte and small files
The inventory contains 242 zero-byte files across 33 papers, all in
`source`, with the empty-content digest `e3b0c442…`. `latex` includes
seventeen of them across seven papers: thirteen `tex`, two `bib`, one `bbl`
and one `cfg`. There are none in `pdf`, `ps` or the indexed HTML family.
`paper_text` and `sample` have no empty or null strings; their minimum
lengths are twelve and 521 characters respectively.
These are separate from non-empty NUL-filled files. Two papers contain both
types: `2405.14882` and `2505.19488`. Testing only `size > 0` excludes
empty files but retains zero-filled ones.
| empty-file classification | rows | papers |
|---|---|---|
| Python `__init__.py` files | 122 | 2 |
| macOS Photos database/index sidecars from one submission | 34 | 1 |
| other files | 86 | 31 |
The paper counts overlap: `2505.19488` has both Photos sidecars and an empty
`images/transformer.pdf`. Its sidecars include thirteen SQLite WAL files,
twelve change journals, three Spotlight shards and two locks. Of 669
`__init__.py` files corpus-wide, 122 are empty and the others reach
104,648 bytes; `2211.10897` contributes 118 of the empty files. An empty
package marker is normal. Other zero-byte files include drafts,
bibliographies, build artifacts, fourteen ancillary CNC toolpaths, four GLM
headers, three Windows `:Zone.Identifier` sidecars and the two PDF-named
source files identified above.
All 242 were checked against original source archives. Of these, 241 were
already empty members of submission tars. For `1810.03044`, arXiv's outer
payload member `1810/1810.03044.gz` in `arXiv_src_1810_009.tar` was itself
empty. No original differed or was missing.
`1810.03044` is the fifth form in the
[withdrawal/removal account](#withdrawal-and-removal-records):
an empty whole submission, with no PDF, PostScript or `paper_text` row.
Metadata identifies an administrative copyright removal. The other 32
papers have both rendered PDFs and `paper_text`; their packages contain
7–3,229 files, with first submissions from 1996-09-13 to 2026-05-26.
| config | smallest non-empty row | contents |
|---|---|---|
| `source` | 1 byte | 2,431 one-byte files, including 1,603 newlines |
| `latex` | 1 byte | subset of source files |
| `ps` | 3 bytes | `%!` plus newline, five copies of one blob |
| indexed HTML family | 9 bytes | `TOC.html\n`, an arXiv `00TOPLEVELFILE` |
| `paper_text` | 12 characters | `%auto-ignore` |
| `pdf` | 322 bytes | a PDF declaring zero pages |
The content-config minima were read from Parquet and re-hashed. The HTML
minimum was read from the held build object because no HTML config exists.
Rendered `v.ps` files have a minimum of 9,241 bytes; none is
below 4,096. All tiny `ps` rows are assets. The 62 assets at most 64 bytes
total 2,823 bytes across eighteen papers: 46 bounding-box sidecars, six
TeX editor-directive files and ten bodyless PostScript headers. All begin
`%!`, illustrating why that prefix alone is not a reliable format test.
The five three-byte objects contain `%!\n`; one is
`hep-lat/9204001`'s `figure1_too_big.ps`. The same blob occurs eight times
across six papers corpus-wide, including three source copies.
All fifteen PDF-config files below 4,096 bytes were read in full and have
headers, end markers and readable page trees. Fourteen declare one page.
`quant-ph/0504128` v1 is a 322-byte, zero-page PDF produced by AFPL
Ghostscript 7.04, with `/Kids [] /Count 0`, a four-object cross-reference
table and `startxref 167`. The same bytes occur in source as
`quant-ph0504128.pdf`. Its metadata says “9 pages, latex, no figures”, but
its version is reported at `0kb`. The next smallest PDF, `1009.4523` v1,
is 714 bytes with one blank page. Small size does not necessarily indicate
a misnamed or truncated file.
HTTP error bodies instead occur in `source`: sixty files across forty
papers, 695 bytes in total. These are 45 copies of `Not Found`, twelve of
`Internal Server Error`, two of `404: Not Found` and one `Not Found` with
a newline, found by digest. Their names include fifteen `.pdf`, thirteen
`.png`, nine `.eps`, eight `.bbl`, five `.jpg`, and one each of `.pptx`,
`.tex`, `.txt`, `.aux`, `.bib`, `.sty`, `.synctex`, `.log`, `.pygtex`, `.zip`.
These files contain saved HTTP error responses. The upstream tool that wrote
them was not identified. None occurs in `pdf`, `ps` or the HTML family.
| config | non-empty rows under 16 B | under 64 B | under 512 B | all rows |
|---|---|---|---|---|
| `source` | 62,789 | 205,132 | 1,528,835 | 47,373,604 |
| `latex` | 21,956 | 46,019 | 383,781 | 12,348,082 |
| `ps` | 6 | 62 | 4,261 | 2,237,839 |
| indexed HTML family | 2 | 73 | 544 | 11,927 |
| `pdf` | 0 | 0 | 1 | 4,973,956 |
Source rows at most 64 bytes, including empty files, total 209,219 across
73,256 papers and 6,891,503 bytes. They include 44,203 TeX-extension files
(11,416 exact `%auto-ignore` markers), 41,878 build artifacts, 20,993
`00README.XXX` directives, 6,552 Subversion metadata files, 2,146 empty
bibliography environments and 1,442 MIME declarations. Sixty-four are
NUL-filled, including the 42 Krita records.
The one-, two- and three-byte files were classified exhaustively by matching
their digests against possible byte strings: 2,431 one-byte rows across
1,622 papers (1,603 newlines), 2,501 two-byte rows across 1,155 papers
(549 double newlines), and 2,225 three-byte rows across 555 papers
(276 triple newlines).
### Nested archives
Nested archives are retained without recursive expansion. The filename-based
set contains 661 files across 496 papers, 1,730,346,673 raw bytes and
580 distinct blobs. It uses `.rar` (279), `.xz` (123), `.7z` (87), `.tar`
(85), `.gz` (41), `.zip` (19), `.zst` (eleven), `.tgz` (six), `.lzma`
(five), `.bz2` (four) and `.z` (one). It excludes 61 `.jar` files and other
containers such as `.sit`, `.egg`, `.deb`, `.cab`, `.lzh`, `.dmg`, `.pkg`.
Of the selected files, 151 are ancillary, containing 787,951,200 bytes
(45.5% of selected archive bytes).
All 661 files were read from Parquet and their headers inspected. The audit
found 555 nonzero containers, plus 68 entirely NUL-filled files and 38 other
non-containers. The 68 contain 78,649,779 bytes (63 `.tar`, five `.tgz`);
52 of them are valid empty archives, so grouping all 106 residual files as
“not archives” would be misleading. The other 38 are thirty xv thumbnails
under `xxx.xvpics/`, three uncompressed TeX files, two PDFs named `.zip`,
one AppleDouble sidecar, one PostScript file named `.Z` and one nine-byte
`Not Found` body named `.zip`. All 106 together contain 83,788,775 bytes
across 77 papers.
Of the 555 nonzero containers, eight differ from their filename format:
five `.zip` files are RAR, one is 7z and two `.tar` files are xz.
## Licensing
Paper licences vary. In the metadata population, 60.42% carry arXiv's
non-exclusive distribution licence and 14.38% have no recorded licence.
The non-exclusive licence grants rights to arXiv; it does not itself grant
downstream redistribution rights. A missing value is not a public-domain
declaration. See [arXiv's licence terms](https://arxiv.org/licenses/nonexclusive-distrib/1.0/)
and this repository's [LICENSE](LICENSE).
The recorded paper-level licence is copied to `versions`; it is not a
per-version licence history. arXiv notes that different revisions can have
different licences. Check the relevant version where that distinction
matters. arXiv separately releases its metadata under CC0, and this
compilation's organisational layer is also dedicated to CC0. Neither
statement relicenses the papers' full text. See the
[arXiv licensing policy](https://info.arxiv.org/help/license/index.html).
## Attribution
Cite the papers your work uses as well as the dataset. The supplied
`arxiv_abs_url` and `arxiv_pdf_url` fields provide links to their arXiv
records and downloads.
```bibtex
@misc{arxiv_complete_2026,
title = {arXiv Complete Corpus},
author = {secemp9},
year = {2026},
note = {Snapshot: metadata 2026-08-30, files 2026-09-05},
url = {https://huggingface.co/datasets/secemp9/arxiv-complete}
}
```
Please also acknowledge arXiv as the source of the material.
Use of its interfaces is governed by the [arXiv API terms](https://info.arxiv.org/help/api/tou.html).
## Maintenance
This release has no automatic refresh. A rebuild would update the snapshot
dates and recompute the version flags.
Report errors, corrections or takedown requests through a discussion on this
repository. Authors requesting removal will have their work removed from the
next revision, as described in [LICENSE](LICENSE).