10 ML papers you should implement at least once

A lot of people read the papers that shaped machine learning, but far fewer sit down and build them, which is a shame because most of them are much smaller than they look. Each of the ten below fits in an afternoon of code, and each one makes far more sense once you have watched it train and seen the numbers move.

1. Word2Vec

A one-hot vector carries no meaning, so "cat" sits exactly as far from "dog" as it does from "carburetor". Word2Vec gives every word a dense vector and trains it to predict the words that appear around it. Skip-gram with negative sampling scores one real pair of words against a few random ones:

[未转换内容，参见原始 JSON]

The input vectors start small and random and the output vectors start at zero, so every dot product begins at 0, σ(0) = 0.5, and each term costs log 2 = 0.69. Training pushes the real pairs up and the random pairs down, and after enough steps the words that share contexts end up with vectors close to each other.

Implement it yourself at tensortonic.com/research?paper=word2vec

2. Adam

Plain gradient descent gives every parameter the same step size, even though some gradients are huge and others are tiny. Adam gives each parameter its own step, built from two running averages of the gradient g:

[未转换内容，参见原始 JSON]

m tracks the direction and v tracks the size, and because both start at zero they underestimate the truth early on. With β₁ = 0.9 and g = 0.5, the first m is only 0.05, a tenth of the real gradient, so Adam divides it by (1 − 0.9) to get 0.5 back and does the same for v with β₂. After that correction the first update is almost exactly lr in size, whatever the size of the gradient was.

3. Batch Normalization

Each layer's inputs keep shifting as the layers before it change, which makes deep networks slow and unstable to train. Batch norm normalizes every feature across the batch and then lets the network rescale it with two learned numbers:

[未转换内容，参见原始 JSON]

For the batch [2, 4, 6], the mean is 4 and the standard deviation is 1.63, so with γ = 1 and β = 0 the output is [−1.22, 0, 1.22]. At inference there may be only one example, so the layer has to use the running mean and variance it saved during training, and forgetting to switch to them is one of the most common batch norm bugs.

4. Dropout

Large networks are very good at memorizing their training data, and dropout fights this by randomly switching off units during training so that no unit can rely on one particular neighbour being there:

[未转换内容，参见原始 JSON]

Here p is the probability of keeping a unit. With p = 0.8, each unit survives 80% of the time and the survivors are scaled by 1 / 0.8 = 1.25, which keeps the expected activation the same as without dropout, so at test time you turn the mask off and change nothing else. The code is about three lines, and most dropout bugs come from leaving it switched on during evaluation.

5. ResNet

Before ResNet, stacking more layers often made a network worse, and the deeper model had higher error even on its own training set, so overfitting could not explain it. ResNet lets each block learn a correction to its input instead of a whole new transformation:

[未转换内容，参见原始 JSON]

The derivative of a block is then ∂F/∂x + 1, and that +1 gives the gradient a direct path back through the network however badly F behaves. If every layer shrank the gradient by half, 20 plain layers would leave 0.5²⁰ ≈ 0.000001 of it, while the skip connections carry the full signal back through all 20.

Implement it yourself at tensortonic.com/research?paper=resnet

6. XGBoost

Gradient boosting builds trees one after another, with each new tree fixing the mistakes of the ones before it. XGBoost uses both the gradient and the curvature of the loss, and it shrinks every leaf toward zero, so the best weight for a leaf is:

[未转换内容，参见原始 JSON]

G is the sum of the gradients in that leaf and H is the sum of the Hessians. With G = −6, H = 3 and λ = 1, the weight is 6 / 4 = 1.5, while without λ it would be 6 / 3 = 2.0, so the regularizer makes each tree take a smaller step.

7. GANs

Writing down the distribution of real images is hopeless, so GANs avoid it by training two networks against each other. The generator turns random noise z into fakes, and the discriminator tries to tell them apart from real data x:

[未转换内容，参见原始 JSON]

When the generator wins completely, the discriminator can't tell real from fake and outputs 0.5 for everything, and the objective settles at 2 · log 0.5 = −1.386. Each network is chasing a target the other keeps moving, which is why GANs are hard to stabilize, and training one on a 1D Gaussian lets you watch the two distributions chase each other on a single plot.

Implement it yourself at tensortonic.com/research?paper=gan

8. Attention Is All You Need

RNNs read one token at a time and gradually lose track of what came early in the sequence. Attention lets every token look at every other token directly:

[未转换内容，参见原始 JSON]

The √d_k is there because dot products grow with dimension. With d_k = 64 and query and key entries of unit variance, the dot product has a standard deviation of √64 = 8, and scores that large saturate the softmax, which then passes back almost no gradient. Dividing by 8 brings the scores back to a standard deviation of 1.

Implement it yourself at tensortonic.com/research?paper=transformer

9. BERT

A left-to-right language model can never use the words that come after the current one, so BERT hides some tokens and predicts them from both sides. It picks 15% of the tokens and then treats them like this:

• 80% are replaced with [MASK]

• 10% are replaced with a random token

• 10% are left unchanged

In a 512-token sequence that is about 77 predictions per pass. The random and unchanged cases matter because [MASK] never appears during fine-tuning, so the model must not learn to pay attention only when it sees that symbol.

Implement it yourself at tensortonic.com/research?paper=bert

10. Language Models are Few-Shot Learners

GPT-3 showed that a large enough decoder-only Transformer can pick up a new task from a few examples in the prompt, with no gradient updates at all. Its size is what made that work, and 175 billion parameters at 2 bytes each in FP16 come to 350 GB, more memory than any single GPU has. You can't train it at home, but you can build a tiny character-level GPT with a causal mask and temperature sampling, put a few examples in its prompt, and see how its output changes. The diagram below is GPT-2, and GPT-3 keeps the same decoder-only design at a much larger size.

Implement GPT-2 yourself at tensortonic.com/research?paper=gpt2

Every idea on this list was invented to fix something that broke without it, and the easiest way to feel that is to delete it from your own code and watch training get worse. TensorTonic walks through each of these step by step and gives you problems where you write the code yourself and run it against tests, and you can start at tensortonic.com.