Today's concept: linear regression from scratch

Strip away the hype and a neural network's simplest form - one neuron, no activation function - is just linear regression wearing a trench coat.

The whole thing is three moves.

1) Draw a line. To predict a house price from its square footage, you fit price = w * size + b. w (the weight) is how steeply price climbs per square foot; b (the bias) is where the line starts at zero size. Two numbers, and that is the entire model.

2) Measure the miss. That is the loss, usually mean squared error: the gap between each prediction and the real price, squared, then averaged. Squaring stops misses above and below the line cancelling out, and makes one big miss hurt more than several small ones.

3) Nudge. Take the derivative of the loss with respect to w and b. That gradient says which direction makes the error worse, so you step the opposite way. How far you step is the learning rate: too small and you crawl, too big and you overshoot. Repeat until the line stops moving.

Here is the twist: linear regression is one of the few models you never need that loop for. It has an exact algebraic answer (the normal equations) you can solve in one shot. So why write the loop by hand? Because the loop generalizes and the algebra does not. Stack layers, add non-linearities, reach billions of parameters, and no formula exists any more.

Which is why you keep meeting it. Backprop is step 3's derivative, chain-ruled back through every layer. AdamW, the default optimizer for transformers, is still just a smarter answer to "how big should the nudge be." Even reasoning models that think longer at inference had every weight placed this way first.

LoRA, the cheap way to fine-tune, freezes the pretrained weights and trains small low-rank matrices instead, running this exact loop (its paper reports up to 10,000x fewer trainable parameters on GPT-3 175B). The learning rate still bites: the best rate shifts with the adapter's rank, which is why rank-stabilized LoRA scales by alpha/sqrt(r) not alpha/r.

And linear regression never retired: interpretability researchers fit a plain linear model on a frozen LLM's activations (a "linear probe") to test whether a concept sits in there in a readable direction.

Guess, measure, nudge. Write it once without a library and the rest stops looking like magic.

Quick check before you scroll: In price = w * size + b, what does gradient descent actually adjust - the data, or something else?

Full breakdown + the answer: frankduah.me/learnings/2026-08-15-linear-regression-from-scratch

New here? I post a bite-size AI / ML concept like this every day - follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH

#LinearRegression #AI #LLM #AIAgents #MachineLearning