Token embeddings and the geometry of meaning

Token embeddings are the lists of numbers that a language model uses in place of words. Before a model can predict or attend to anything, text is cut into tokens and each token is mapped to a vector, so that meaning becomes a matter of position and direction in a space with hundreds or thousands of dimensions. This page serves the futurism goal of this wiki, along with learning and research. It completes a short series on the mathematics of language models, with Next-token prediction and cross-entropy loss, Attention in transformers explained with numbers, and How central gradient descent is to modern AI. It covers how text is split, how vectors come to carry meaning, what the famous king and queen example really shows, and why models stumble on questions as simple as counting letters.

From text to tokens

Models do not read words or letters directly. They read tokens, which are often pieces of words. Most modern tokenizers descend from byte pair encoding (BPE) as adapted by Sennrich, Haddow, and Birch in 2016. Their method starts with single characters and repeatedly finds "the most frequent pair" of symbols and merges it into a new symbol, so common words become one token while rare words are spelled out in pieces. The goal was to handle "rare and unknown words as sequences of subword units."

The result can be surprising. Running OpenAI's open source tiktoken library on the word "strawberry" gives these splits:

tokenizer      "strawberry"            " strawberry" (with a leading space)
gpt2           st | raw | berry        one token
cl100k_base    str | aw | berry        one token
o200k_base     st | raw | berry        one token

With a leading space, as the word usually appears mid-sentence, it is a single token. The model never sees the letters r, r, and r one at a time.

Tokenization also has a fairness cost. Petrov and colleagues found that the same text translated into different languages can differ in token length "up to 15 times in some cases," which affects the price of commercial AI services, response time, and how much text fits in the context. For a future where AI tutors serve every language, that is a design problem worth solving.

From tokens to vectors

Each token ID selects one row of a large table, the embedding matrix. That row is the token's vector. The table starts as random numbers and is adjusted by gradient descent along with the rest of the model, so tokens that are used in similar ways end up near each other. This follows an old idea from linguistics. Jurafsky and Martin describe the distributional hypothesis, formulated in the 1950s by Joos, Harris, and Firth: "Words that occur in similar contexts tend to have similar meanings."

Closeness is usually measured by cosine similarity, the cosine of the angle between two vectors: 1 means the same direction, 0 means unrelated directions. Here is an illustrative example with made-up 3-dimensional vectors.

king  = (0.9, 0.6, 0.4)     man   = (0.1, 0.6, 0.9)
queen = (0.8, 0.1, 0.4)     woman = (0.1, 0.2, 0.9)

king - man + woman = (0.9, 0.2, 0.4)

cosine similarity to (0.9, 0.2, 0.4):
  queen 0.995    king 0.940    woman 0.526    man 0.522

In this toy, queen wins. Notice that king itself is a close second. That detail matters below.

What king minus man plus woman really shows

In 2013 Mikolov and colleagues introduced word2vec, which learned word vectors from a 1.6 billion word data set in less than a day. They reported that "vector('King') - vector('Man') + vector('Woman') results in a vector that is closest to the vector representation of the word Queen," building on a companion paper on linguistic regularities in word vectors.

There is a catch. Nissim, van Noord, and van der Goot showed that the standard analogy code "explicitly prevents" returning any of the input words. When they allowed input words, accuracy on the Google analogy test set fell from 0.74 to 0.21 for the usual method, mostly because the answer to "man is to king as woman is to X" became king. The vectors do hold real structure, but the famous demonstration was helped by a rule that removed the most obvious wrong answer.

Bias lives in the geometry

Because embeddings learn from human text, they absorb human stereotypes. Bolukbasi and colleagues found that vectors trained on Google News "exhibit female/male gender stereotypes to a disturbing extent," and that gender bias was "captured by a direction in the word embedding." They proposed removing that direction. Gonen and Goldberg then showed that such removal is "mostly hiding the bias, not removing it," since the information can still be recovered from distances between words. Nissim and colleagues add that some widely shared biased analogies were partly an artifact of the same exclusion rule; with it removed, "man is to doctor as woman is to" returns doctor. Bias is real, and measuring it well takes more than analogies.

Static versus contextual embeddings

Word2vec gives each word one fixed vector, so "bank" has the same vector by a river and in a city. Jurafsky and Martin call these static embeddings. Modern models start from a static table and then let attention layers rewrite each vector using its context. Ethayarajh measured how far this goes: in ELMo, BERT, and GPT-2, "less than 5% of the variance in a word's contextualized representations can be explained by a static embedding for that word." In a large model, the input embedding is a starting point, not the final meaning.

Position

Attention by itself treats its input as an unordered set, so models add information about word order. The original transformer added fixed patterns built from "sine and cosine functions of different frequencies" to each embedding. Many current models use rotary position embedding (RoPE), which "encodes the absolute position with a rotation matrix" so that attention scores depend on the relative distance between tokens.

Why counting letters is hard

The "strawberry" split above is the usual explanation for why chatbots have miscounted the letter r in that word. Fu and colleagues tested this directly across many models and words. They found that models "are capable of recognizing the letters but not counting them," and that most models could not correctly count a letter that appears more than twice in a word. So tokenization hides the letters, but the deeper weakness is counting. Written with spaces, as "s t r a w b e r r y," the same word becomes ten tokens in cl100k_base, one per letter, which gives the model separate items to count.

Teaching and learning with embeddings

Tokenize your own name. Install tiktoken and run the line below on names, slang, and words from other languages. Students see quickly which words the tokenizer treats as common.

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
print([enc.decode([t]) for t in enc.encode("strawberry")])

Build BPE by hand. Write a short sentence on the board, count adjacent letter pairs, merge the most common pair, and repeat five times. Andrej Karpathy's minbpe repository offers the same algorithm in short, readable code.

Explore a real space. The TensorFlow Embedding Projector lets learners search a word and see its nearest neighbors in a trained embedding, which makes the toy arithmetic above feel real.

Discussion questions, essay ideas, and learning related AI prompt ideas

• If meaning is a position in a vector space, what parts of meaning does that picture capture well, and what does it leave out?
• The king and queen analogy worked partly because the code banned the input words. How often do famous demonstrations in science depend on a hidden rule like this, and how can learners spot one?
• Some languages need many times more tokens than English for the same text. What would a fair tokenizer look like, and who should pay for building one?
• Removing a bias direction from embeddings mostly hid the bias. What does that suggest about fixing bias in larger AI systems trained on human writing?
• Why can a model that writes fluent essays fail to count the letter r in "strawberry," and what does that say about the difference between knowing a word and knowing its spelling?
• How should a teacher explain contextual embeddings to a student who already understands that one word can have several meanings in a dictionary?
• Use an embedding tool to find three surprising nearest neighbors for a word you know well, and explain what training text could have caused each one.

See also

• Next-token prediction and cross-entropy loss
• Attention in transformers explained with numbers
• How central gradient descent is to modern AI
• Artificial intelligence
• Natural language processing
• Linguistics
• Language
• Words
• Mathematics
• Learning