niels-gpt-1: a language model in two days

Two days from first commit to published checkpoint: 256 tokens of vocabulary, six hours of training, one laptop.

repo

niels-gpt-1 is a byte-level GPT, written and trained from scratch on a MacBook Air. It came out of makemore, two November days of Karpathy's zero-to-hero exercises: character-level name models and a hand-written autograd class.

The charter fixes the goal and the box — "i write the transformer code, i train the weights, and i can explain every part"; one M4 laptop on MPS; a training run under six hours; a byte-level tokenizer. Two days separate the first commit, 2025-12-14, from the published checkpoint, whose best.pt is dated 2025-12-16.

The tokenizer was the hard call. Subwords from day one were the alternative; the charter pins bytes — vocab 256, every byte a token. A tokenizer I did not build would have been the one part I could not explain. The cost is context: T=384 tokens is 384 bytes, roughly a hundred words of memory.

The rest is deliberately legible. RoPE implemented by hand, with its own PR and tests. A decoder-only transformer with LayerNorm, GELU MLP, and weight tying, at V=256, T=384, C=384, six layers, four heads. Pretraining mixes wikitext-103, my Roam notes, and a hand-written primer at 0.72, 0.18, and 0.10. A chat CLI with temperature and top-k sampling and stop sequences. Attention stays a manual softmax, and a traced forward path returns the probability rows niels-gpt-app displays.

Numbers, by wc -l and grep over tracked files: 5,720 lines of Python across 43 files; 107 test functions in 16 files; the dependencies are torch and datasets.

What is wrong with it. No loss curve, no eval number, and no sample transcript is committed anywhere, so I can cite the machinery and not the model. The README's caveats contradict the config in one place — "context is 512 tokens" against a T=384 default. Best-checkpoint selection keys on wikitext loss alone. And the repo is a frozen snapshot: the original repo was renamed and rewritten in place as niels-gpt-2, the two share their first 29 commits, and v1's final commit is a README todo recording the link debt the split created.

The byte-level constraint was the point. niels-gpt-2 removes it.