MATH 170 · From words to numbers · One-Hot & Embeddings
One-hot can’t say “alike” embeddings · cosine
You are the AI. Before you can compute with words you have to turn
them into numbers. Here are the two classic ways — one where every word is a stranger to
every other word, and one where the map itself knows that cats and
dogs go together.
In one-hot land, cats is exactly as far from dogs as from rose.
Identity, not meaning — every word’s ID card differs from every other’s
in exactly two slots, so every pair ties.
“I like cats and dogs” — the worksheet sentence, encoded
Five words → five one-hot rows in ℝ⁵ — the sentence is just a stack of
ID cards. Check any two different words: still exactly 2 apart, “cats”
and “dogs” included. The rows store order; they store no
meaning.
Each word owns one slot: a 1 in its own column, 0 everywhere else — an
identity matrix. (The worksheet’s own version is the 3-word table
{cats, dogs, flowers} in ℝ³ — same picture, smaller.) Tap two chips to
measure a pair.
Where do these coordinates come from?
Not from a model. They are illustrative — we chose them by hand, so that related
words sit near each other and the arithmetic below comes out at readable numbers.
A real model is given nothing but text and learns its coordinates by
being wrong over and over and nudging them; that is the next activity. What is honest here
is everything downstream: once a word has a pair of numbers, the dot product, the lengths
and the cosine are computed exactly as they would be on real ones.
Drag any word to a new spot — you be the training:
dragging IS the stand-in for training. Real training nudges these positions from
data; here your hand does it, and every similarity reacts live.
the worksheet’s 5-word sentence → 5 one-hot rows
u·v and the lengths ‖u‖, ‖v‖ are exactly the dot product and vector
length you already know. Dividing the dot product by both lengths cancels how long the
arrows are, so the only thing left is direction:
cos θ = −1 opposite · 0 unrelated (a right angle) · +1 same
direction. That is why two words can be far apart on the map yet still point the
same way — cosine reads the angle, not the distance.
Try: check three pairs in one-hot world, then flip worlds and drag
lion into the flower patch — watch cos(lion, cats) fall.