01
2026Author
Glassbox
A transformer from scratch, every gradient derived by hand
A working transformer implemented in NumPy with no autograd anywhere. It supports both GPT-2 and Llama-style architectures (RMSNorm, RoPE, SwiGLU, and a KV-cache), and every forward and backward pass is hand-derived and gradient-checked to 1e-8. It trains and generates text on a plain CPU.
- Gradient check
- 1e-8
- Frameworks
- None (NumPy)
- Architectures
- GPT-2 · Llama
- Runs on
- CPU
- Python
- NumPy