Next-token prediction and cross-entropy loss
Next-token prediction is the training goal behind nearly every large language model: given some text, assign a probability to every possible next token, and get scored on how much probability went to the token that actually came next. The score is called cross-entropy loss. This page serves the futurism goal of this wiki, along with learning and research. It is the first of a short series on the mathematics of language models, together with Attention in transformers explained with numbers, Token embeddings and the geometry of meaning, and How central gradient descent is to modern AI. The question here is concrete: how does an objective as narrow as guessing the next word produce translation, question answering, and code, and where does that explanation stop working?
The objective in one line
A model reads tokens x1, x2, and so on up to xt, and outputs a probability for every token in its vocabulary as the candidate for position t+1:
P(x_(t+1) | x_1, ..., x_t)
The model does not output probabilities directly. Its last layer produces one raw score, called a logit, per vocabulary entry. The softmax function turns those scores into probabilities by exponentiating each one and dividing by the total, so every value is positive and they sum to 1. The Deep Learning textbook puts it simply: "Any time we wish to represent a probability distribution over a discrete variable with n possible values, we may use the softmax function."[1]
During training, the loss for one position is the negative logarithm of the probability given to the correct token:
L = -log P(correct token | context)
Averaged over trillions of positions, this is the number that gradient descent pushes down, with backpropagation supplying the gradients.
A worked example
The numbers below are illustrative, not output from a real model. Suppose the context is "The capital of France is" and the model has four candidates.
token logit softmax probability loss if correct (natural log) Paris 3.8 0.843 0.17 Lyon 1.5 0.084 2.47 London 0.8 0.042 3.17 other 0.5 0.031 3.47
If the text really continues with "Paris," the loss is small: -ln(0.843) is about 0.17. If the text continues with "Lyon," perhaps because the sentence was about a regional capital, the loss is about 2.47, more than fourteen times larger. The logarithm makes the penalty grow without limit as the probability of the right answer approaches zero, so a model is punished hardest for being confidently wrong.
Perplexity is the same quantity on a friendlier scale. Jurafsky and Martin define it as "the inverse probability of the test set" normalized by the number of words or tokens.[2] In practice it equals e raised to the average loss. If a model gives the correct tokens of a three-token text probabilities of 0.84, 0.5, and 0.2, the average loss is about 0.83 and the perplexity is about 2.3. That reads as "the model was, on average, about as unsure as if it were choosing among 2.3 equally likely options." A model that guessed uniformly from a 50,000-token vocabulary would have a loss of about 10.8 and a perplexity of 50,000.
Why a narrow objective produces broad abilities
Shannon saw prediction as a measure of knowledge. In 1948 Claude Shannon generated text from letter and word statistics, from a "first-order word approximation" to a "second-order word approximation," and each step looked more like English.[3] In 1951 he asked people to guess a passage letter by letter. His method "exploits the knowledge of the language statistics possessed by those who speak the language," and he estimated that long range effects bring ordinary English down to something like one bit per letter.[4] The insight carries over: to predict well, you must know whatever makes the next symbol predictable. For the sentence about France, that is geography. For the next line of a program, it is the logic of the code.
The tasks are already in the text. The GPT-2 paper argued that language models "begin to learn these tasks without any explicit supervision" when trained on millions of web pages, because the web already contains translations, answers, and summaries written in ordinary prose. Its title stated the claim: language models are unsupervised multitask learners.[5] GPT-3 pushed this further. With 175 billion parameters, it was given tasks and a few examples "purely via text interaction" and "without any gradient updates or fine-tuning," and it did well on translation, question answering, and 3-digit arithmetic.[6]
Lower loss tracks scale predictably. Kaplan and colleagues found that cross-entropy loss "scales as a power-law with model size, dataset size, and the amount of compute," with some trends spanning more than seven orders of magnitude.[7] Hoffmann and colleagues later trained over 400 models and concluded that model size and training tokens "should be scaled equally"; their 70 billion parameter Chinchilla beat the much larger Gopher.[8] This predictability is why the objective matters for futurism: labs plan budgets around loss curves.
Where the story breaks
Training and use differ. In training, the model always sees the true previous tokens. In use, it conditions on its own output. Bengio and colleagues described how this "discrepancy between training and inference can yield errors that can accumulate quickly along the generated sequence."[9] Likelihood also makes a poor decoding goal: always picking the most probable token yields text that is "bland and strangely repetitive," which led to sampling methods such as nucleus sampling.[10]
Calibrated guessing produces hallucinations. Kalai and colleagues argue that calibration, and with it some rate of error, is "a natural consequence of the standard cross-entropy objective." A model that matches the statistics of its data must sometimes produce plausible falsehoods. Their example: if 20% of birthday facts appear exactly once in the pretraining data, then base models should be expected to hallucinate on at least 20% of birthday facts. They add that benchmarks scored as right or wrong reward guessing over saying "I do not know."[11]
Chat assistants are trained on more than next-token loss. The InstructGPT paper says that predicting the next token on a web page "is different from the objective" of following instructions helpfully and safely, and it fine-tuned GPT-3 with human demonstrations and reinforcement learning from human feedback (RLHF). Its 1.3 billion parameter model was preferred to the 175 billion parameter GPT-3.[12] The change has a cost. The GPT-4 report notes that the pretrained model "is highly calibrated," while "post-training hurts calibration significantly."[13] So the assistant you talk to is a next-token predictor that has been pushed away from pure prediction.
Teaching and learning with next-token prediction
A Shannon game. Cover a short passage and have the class guess one letter, then one word, at a time, recording the number of guesses needed. Students quickly notice that later words are easier, and that guessing well requires knowing the topic. That is the core of language model training, done by people.
A tiny calculation. This Python snippet turns logits into probabilities and losses. Students can change the logits and see that raising one score lowers the others.
import math
logits = {"Paris": 3.8, "Lyon": 1.5, "London": 0.8, "other": 0.5}
total = sum(math.exp(v) for v in logits.values())
for tok, v in logits.items():
p = math.exp(v) / total
print(tok, round(p, 3), "loss", round(-math.log(p), 3))
A habit for learners. Predicting before reading, then checking, is a sound study practice in its own right. It turns reading into a stream of small, scored tests, which is what the loss function does for a model.
Discussion questions, essay ideas, and learning related AI prompt ideas
- Shannon measured how predictable English is by asking people to guess letters. What would a modern version of his experiment reveal about how people and language models differ?
- If a model must understand chemistry to predict a chemistry textbook, is there any limit to what next-token prediction could teach a model, and what evidence would show that limit?
- Why does the logarithm in cross-entropy loss punish confident wrong answers so much more than unsure ones, and is that a good rule for grading students too?
- Kalai and colleagues argue that right-or-wrong benchmarks reward guessing. How should tests for both people and AI systems be redesigned to reward honest uncertainty?
- Does fine-tuning with human feedback make a model more useful, less truthful, or both, and how could someone check this for a model they use?
- Scaling laws let labs predict loss before training. What would it mean for planning research, education, and the economy if those curves keep holding, or suddenly stop?
- Design a 30-minute lesson that teaches perplexity using only a deck of cards and a short paragraph of text.
See also
References
- ↑ Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. "Deep Feedforward Networks," chapter 6, section 6.2.2.3 "Softmax Units for Multinoulli Output Distributions," in Deep Learning. MIT Press, 2016. https://www.deeplearningbook.org/contents/mlp.html
- ↑ Jurafsky, Daniel, and James H. Martin. "N-gram Language Models," chapter 3 in Speech and Language Processing, draft of August 19, 2026. https://web.stanford.edu/~jurafsky/slp3/3.pdf
- ↑ Shannon, Claude E. "A Mathematical Theory of Communication." Bell System Technical Journal, vol. 27, 1948, pp. 379 to 423 and 623 to 656. https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf
- ↑ Shannon, Claude E. "Prediction and Entropy of Printed English." Bell System Technical Journal, vol. 30, no. 1, January 1951, pp. 50 to 64. https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf
- ↑ Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. "Language Models are Unsupervised Multitask Learners." OpenAI, 2019. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
- ↑ Brown, Tom B., et al. "Language Models are Few-Shot Learners." arXiv:2005.14165, 2020. https://arxiv.org/abs/2005.14165
- ↑ Kaplan, Jared, et al. "Scaling Laws for Neural Language Models." arXiv:2001.08361, 2020. https://arxiv.org/abs/2001.08361
- ↑ Hoffmann, Jordan, et al. "Training Compute-Optimal Large Language Models." arXiv:2203.15556, 2022. https://arxiv.org/abs/2203.15556
- ↑ Bengio, Samy, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. "Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks." arXiv:1506.03099, 2015. https://arxiv.org/abs/1506.03099
- ↑ Holtzman, Ari, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. "The Curious Case of Neural Text Degeneration." arXiv:1904.09751, 2019. https://arxiv.org/abs/1904.09751
- ↑ Kalai, Adam Tauman, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. "Why Language Models Hallucinate." arXiv:2509.04664, 2025. https://arxiv.org/abs/2509.04664
- ↑ Ouyang, Long, et al. "Training language models to follow instructions with human feedback." arXiv:2203.02155, 2022. https://arxiv.org/abs/2203.02155
- ↑ OpenAI. "GPT-4 Technical Report." arXiv:2303.08774, 2023. https://arxiv.org/abs/2303.08774