---
title: gpt2_1558M_final4_hf
canonical_url: "https://www.modelscope.cn/models/AI-ModelScope/gpt2_1558M_final4_hf"
md_url: "https://www.modelscope.cn/models/AI-ModelScope/gpt2_1558M_final4_hf.md"
repository: AI-ModelScope/gpt2_1558M_final4_hf
last_updated: 2025-10-20
pipeline_tag: text-generation
tasks:
  - text-generation
model_type:
  - gpt2
architectures:
  - GPT2LMHeadModel
parameters: 1.6B
tensor_type:
  - BF16
library_name:
  - safetensors
  - pytorch
frameworks:
  - pytorch
inference_backends:
  - "deploy_task text/emb"
  - "sglang 0.5.2"
  - "vllm 0.9.2"
downloads: 84
stars: 0
---

# gpt2_1558M_final4_hf

> gpt2_1558M_final4_hf - AI-ModelScope 在 ModelScope 开源的模型。This is a GPT-2 model trained in llm.c for 330K steps (of 1M batch size) on FineWeb-EDU.

AI-ModelScope/gpt2_1558M_final4_hf 是 ModelScope 魔搭社区上的 1.6B 参数text-generation模型，可用 deploy_task text/emb、sglang 0.5.2、vllm 0.9.2 部署。

- **Repository**: AI-ModelScope/gpt2_1558M_final4_hf
- **Tasks**: text-generation
- **Parameters**: 1.6B
- **Inference backends**: deploy_task text/emb, sglang 0.5.2, vllm 0.9.2
- **Downloads**: 84
- **Stars**: 0
- **Last updated**: 2025-10-20

Source: https://www.modelscope.cn/models/AI-ModelScope/gpt2_1558M_final4_hf

---

This is a GPT-2 model trained in llm.c for 330K steps (of 1M batch size) on FineWeb-EDU.

A lot more detailed information is here: https://github.com/karpathy/llm.c/discussions/677 . 

This model has a bit of a complicated history. I wanted to train it for 400K steps, i.e. (`-x 400000`), but it became unstable later in training and exploded around step 330K. Because I was losing my computing quota shortly, I decided to just rewind back to checkpoint 300K, and then instead of going all the way to 400K I started annealing linearly down to 330K. This went without incident and produced this model.

This is the longest I've trained a GPT-2 model for, and it reaches HellaSwag of 62.7 by the end.