2026Author

Glassbox

Most people who use transformers have never watched one learn, because autograd hides the one part that actually teaches you how the model works: the gradients.

The result

A working transformer in NumPy where every forward and backward pass is hand-derived and gradient-checked to 1e-8, trained and generating text on a plain CPU.

Glassbox is a transformer built from scratch in Python with NumPy and nothing else. It supports both GPT-2 and Llama-style architectures, so RMSNorm, RoPE, SwiGLU, and a KV-cache are all written out by hand. There is no autograd anywhere: every derivative in the backward pass was worked out on paper and then verified numerically.

Gradient check: 1e-8
Frameworks: None (NumPy)
Architectures: GPT-2 · Llama
Runs on: CPU

Architecture

Input tokens

Embedding

token + RoPE positions

Transformer blocks

RMSNorm, attention, SwiGLU, KV-cache

Output logits

Cross-entropy loss

Hand-derived backward pass

Gradient check

matches finite differences to 1e-8

Weight update

The weight update feeds back into the blocks each training step, so the same code both trains and generates.

The forward half is the standard transformer flow: tokens become embeddings with RoPE positions, pass through stacked blocks (RMSNorm, attention, SwiGLU, and a KV-cache for generation), and come out as logits scored against a cross-entropy loss. The interesting half is the return trip. Instead of calling a framework's backward method, I derived the gradient of every operation by hand and implemented each one, then checked each against a finite-difference estimate. Only once the analytic and numeric gradients agreed to 1e-8 did a layer count as done.

Key decisions and trade-offs

Hand-derive backprop instead of leaning on autograd

Autograd is the right tool in production, but it also means you can ship a model without ever understanding why a gradient vanishes or explodes. Deriving every backward pass myself, then gradient-checking it to 1e-8, turned the chain rule from something I trusted into something I can reproduce from memory. That is the whole point of the project: the box is glass.

One codebase, two architectures

Supporting both GPT-2 and Llama-style stacks (RMSNorm and RoPE and SwiGLU next to the older LayerNorm and learned-position design) forced the components to be genuinely modular rather than a single hardcoded model. It also made the differences between the two families concrete instead of abstract.

NumPy only, CPU only

No deep-learning framework, no GPU, no CUDA to hide behind. The constraint keeps the code readable and portable, and it proves the ideas do not need an accelerator to be correct. The trade-off is speed, which is fine for a project whose goal is understanding, not throughput.

Results