Building an AI from the ground up, one vector at a time

Everything becomes a number

The first move is that a computer never reads a word; it only does arithmetic. So the first job is turning 'cat' into numbers: a vector — a list of maybe 768 numbers you can picture as an arrow from the origin out to a point.

The trick that makes it useful is that the numbers are chosen so similar meanings land as nearby arrows. 'Cat' and 'dog' point almost the same way while 'helicopter' points elsewhere. Meaning becomes geometry. Nobody hand-picks those numbers; they are learned from a simple pressure, that words showing up in similar sentences get pushed toward similar directions.

The 2013 word2vec demo made it concrete: king minus man plus woman lands nearest to queen, because the man-to-woman step is a consistent direction you can walk from anywhere. A finance reader already meets this idea as cosine similarity between two return series.

The dot product, the one operation under all of it

If meaning is direction, you need to ask how aligned two arrows are, and that is the dot product: multiply two number-lists position by position, then add up the result. One cheap multiply-and-add secretly measures agreement.

Aligned arrows give a big positive number, unrelated ones sit near zero, opposed ones go negative. That single number is a similarity score. The sentence worth keeping: semantic search is dot products. Every time a vector search pulls back a passage, it is comparing embedding arrows with this exact operation.

When people say GPUs matter for AI, this multiply-and-add run billions of times a second is mostly what the GPUs are doing.

The machine that reshapes space

Words-as-arrows need a machine to act on them, and that machine is a neural network built from one small formula, h = f(w·x + b). Here x is the input arrow, w is a set of weights scoring how much the input matches a pattern (the dot product again), b shifts the threshold up or down, and f is a small bend applied at the end.

Stack a row of these into a layer, feed one layer's output into the next, and you get an MLP (multilayer perceptron). Each layer stretches, rotates and folds the space until things that were hopelessly tangled, like sarcastic reviews mixed in with sincere ones, become cleanly separable.

The whole network's knowledge lives in those weight numbers and nowhere else. Depth is not magic; depth is folding space more times.

The one function choice that froze a field

The small bend f looks like a footnote and decided about a decade of progress. Without it, stacking layers is pointless: a stack of straight-line operations collapses into one straight line, so the bend — the activation function — is what makes the folding possible.

The old choice, the sigmoid, squashed every number into 0-to-1 on an S-curve, but the edges of that curve go flat, and flat means a near-zero slope. Since learning passes slopes backward through the layers and multiplies them along the way, ten flat layers multiply ten tiny numbers and the signal shrinks to nothing before it reaches the front. That is the vanishing gradient, and it wrote neural nets off as a dead end for two decades.

The fix, ReLU, is almost insulting: output zero for a negative input, pass a positive input through untouched, so the slope stays exactly 1. Any developer will recognise the lesson — the breakthrough was deleting the thing quietly killing you rather than adding cleverness.

What 'wrong' means, and why models sound sure when they are not

Before the machine can improve, 'wrong' has to become a single number, and for language models that number is cross-entropy: L = minus log P(correct next word). The model reads some text, outputs a probability for every word that could come next, and the loss takes the probability it gave the word that actually came next, takes its log, and flips the sign.

The log turns confident-and-wrong into catastrophe (probability 0.01 costs a loss of 4.6) while an honest hedge is punished gently. The deeper point should reassure anyone from a finance or statistics background: minimising this loss is identical to maximum likelihood, the century-old move of picking the parameters under which the observed data was most probable. GPT training is the most classical thing in statistics wearing new clothes.

It also explains why a model sounds confident when it is wrong: right and wrong answers come out of the same probability distribution wearing the same fluency.

Learning is rolling downhill

The finale is that learning means making the wrongness number go down. Picture the billions of weights as one location in a foggy landscape where altitude is the loss; you cannot see the valley, you can only feel which way is downhill and take a step.

'Which way is downhill' is the gradient, computed by backpropagation, which for all its reputation is the calculus chain rule with careful bookkeeping. The one line the modern world runs on is theta becomes theta minus eta times the gradient: nudge every weight a little against the uphill direction, where eta sets how big a step to take. Run that trillions of times over trillions of words and random numbers become GPT.

The cliffhanger: everything so far gives a word one fixed arrow, so 'river bank' and 'cash from the bank' share the same arrow. Language needs the arrow for 'bank' to bend with its neighbours, and the 2017 fix — every word asking every other word 'are you relevant to me right now', scored by the same dot product — is attention, the transformer Part B promises.

Vocabulary

• dot product — Multiply two number-lists slot by slot and add up; scores how aligned they are.
• MLP (multilayer perceptron) — Layers of simple scoring formulas stacked so each reshapes the data for the next.
• activation function — The small bend added after each layer; without it, all layers collapse into one.
• ReLU (Rectified Linear Unit) — The simplest useful bend: zero for negatives, pass positives through unchanged.
• cross-entropy — The score for how wrong a next-word guess was; confident-and-wrong is punished hardest.
• gradient descent — Feel which way lowers the error, take a small step, repeat until it stops improving.
• attention — Each word asks every other word how relevant it is, so meaning bends with context.

If you're building — what to watch for

• If you run semantic search or RAG, the retrieval step is this dot product and nothing more. A query that misses means the arrows did not align, so rephrase it to say the meaning rather than the remembered keywords.
• Confidence is a style, not a signal. Fluent, sure-sounding output comes from the same distribution as the correct output, so build a verification step instead of trusting tone.
• Flip the minus to a plus in the update rule and gradient descent becomes gradient ascent. It is the same machinery behind any loop that climbs a score, which is worth knowing before you design an eval or a self-improving loop.
• The ReLU story is a debugging heuristic: the breakthrough was deleting the thing quietly killing the signal, not adding cleverness on top. Look for what is silently flattening your pipeline before you add another layer to it.

Reading it critically

• It is Part A only. Attention and the transformer, the thing actually behind ChatGPT, are deferred to Part B, so you get the foundation without the payoff yet.
• word2vec's 'one fixed arrow per word' is the OLD model, and the piece flags this itself in the 'bank' crack. Do not walk away thinking modern LLMs use static embeddings, because they bend the arrow with context.
• 'Confidence is a style, not a signal' is a sharp heuristic but slightly overstated: a well-calibrated model's probabilities do carry real signal. The honest version is that fluency is not correctness, so verify what matters.
• Strongest case against the frame: 'everything is just a vector and a dot product' is true at the base but hides the parts that took the most work, namely tokenization, positional encoding, and how embeddings are actually trained. Clean pedagogy, not the full pipeline.