How central gradient descent is to modern AI

Gradient descent is the method that adjusts the parameters of almost every modern artificial intelligence model, from image classifiers to large language models. This page serves the futurism goal of this wiki, along with learning and research. It asks a specific question: is gradient descent the most important algorithm in AI, more pivotal than backpropagation, attention, or search? The short answer is that gradient descent, paired with backpropagation, is the learning engine of nearly all current AI. It is necessary, but it was never sufficient, and the evidence for that comes partly from the people who made it work.

What gradient descent does

A neural network makes a prediction, a loss function scores how wrong the prediction was, and gradient descent nudges every parameter a small step in the direction that reduces the loss. Repeat that millions or billions of times and the network improves. In practice the gradient is estimated from a small random batch of examples rather than the whole dataset, which is called stochastic gradient descent (SGD).

The standard textbook Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville opens section 5.9 with this sentence: "Nearly all of deep learning is powered by one very important algorithm: stochastic gradient descent (SGD)."

Popular optimizers such as Adam are members of the same family. Kingma and Ba describe Adam as "an algorithm for first-order gradient-based optimization of stochastic objective functions." It still follows the gradient; it only adapts the step size for each parameter.

Gradient descent and backpropagation are different things

People often blur the two. Backpropagation computes the gradient: how much each weight contributed to the error. Gradient descent uses that gradient to change the weights. The same textbook is explicit: "back-propagation refers only to the method for computing the gradient, while another algorithm, such as stochastic gradient descent, is used to perform learning using this gradient." It adds that backpropagation can compute derivatives of almost any function, not only neural networks. Today that idea lives on as automatic differentiation, the feature in tools such as PyTorch and JAX that computes gradients for any program a researcher writes.

So the honest unit of comparison is a pair: backpropagation tells the model which way is downhill, and gradient descent takes the step.

An old idea that waited for its moment

Gradient descent is old. Augustin-Louis Cauchy described the gradient method in a note to the French Academy of Sciences on October 18, 1847, motivated by heavy astronomical calculations. In 1951 Herbert Robbins and Sutton Monro published "A Stochastic Approximation Method," the mathematical ancestor of using noisy estimates in place of exact ones. In 1986 David Rumelhart, Geoffrey Hinton, and Ronald Williams popularized backpropagation for training networks with hidden units, a procedure that "repeatedly adjusts the weights of the connections in the network so as to minimize" the error.

Yet the deep learning boom came decades later. The Deep Learning textbook says the core ideas "have not changed substantially since the 1980s" and that most of the gain from 1986 to 2015 came from larger datasets and much larger networks running on more powerful computers, plus a small number of algorithmic changes. AlexNet, the 2012 image classifier co-authored by Ilya Sutskever, was trained with stochastic gradient descent for five to six days on two NVIDIA GTX 580 GPUs. The algorithm was old. The data, the hardware, and the know-how were new.

What Sutskever actually said

Sutskever, an OpenAI co-founder who later founded Safe Superintelligence, is often linked to this question. No source found for this page shows him calling gradient descent the single most important algorithm in AI. His recorded remarks point the other way. On the No Priors podcast in November 2023, explaining AlexNet, he listed a big network, a large dataset, and the compute to train it, then said of gradient descent: "Gradient descent was around for a long time. It was certain technical insights about how to make it work."

On the Dwarkesh Patel podcast in March 2023 he described the most fundamental advance of deep learning as the realization that "a big neural network trained in backpropagation can do a lot of things," adding that the novelty was neither in the network nor in backpropagation but in seeing what they could do. Asked about Hinton's forward-forward algorithm, he said that for engineering a good system "there is no reason to not use backpropagation," calling it "the only algorithm."

The surprise: a greedy, local method works on huge networks

Gradient descent only looks at the slope where it stands. For a large network the loss surface is wildly non-convex, and for years many researchers expected the method to get stuck in poor local minima. Dauphin and colleagues argued in 2014 that in high dimensions the larger difficulty is "the proliferation of saddle points, not local minima," flat regions that slow learning and "give the illusory impression" of a minimum. Zhang and colleagues then showed that image networks trained with stochastic gradient methods "easily fit a random labeling of the training data," which means overparameterized networks are easy to optimize, and that the open puzzle is why they generalize at all. This is still an active research area, and a good one for students who like mathematics.

Comparing the candidates fairly

• Backpropagation and automatic differentiation. Gradient descent is useless at scale without an efficient way to get gradients for billions of weights. A strong case can be made that this is the co-equal winner.
• Attention and the transformer. The 2017 transformer architecture made language models far more parallel and capable. It decides what is learned, but it is still trained by gradient descent.
• Search and reinforcement learning. AlphaGo combined Monte Carlo tree search with policy and value networks, and the paper reports training those networks with stochastic gradient ascent and descent. Search added lookahead the networks lacked, but gradient descent built the judgment that the search relied on.
• Gradient-free methods. Evolution strategies skip backpropagation entirely. A 2017 OpenAI paper, with Sutskever as a co-author, showed them to be "a viable solution strategy" for reinforcement learning tasks that scales across over a thousand parallel workers. So gradients are dominant, but they are not the only way to learn.

Rich Sutton's essay "The Bitter Lesson" offers a frame for all of this. He argues that "general methods that leverage computation are ultimately the most effective," and that "the two methods that seem to scale arbitrarily in this way are search and learning." Gradient descent won because it is the simplest learning method that keeps improving as compute grows.

Verdict

If the question is "which algorithm does nearly every modern AI system run during training," the answer is gradient descent, with backpropagation computing its gradients. No other candidate is so universal. If the question is "which idea caused the AI boom," the answer has to be a combination: gradient descent, backpropagation, large datasets, GPUs, architectures such as the transformer, and the practical know-how Sutskever described. Gradient descent is the engine, not the whole vehicle. For futurism, that matters: the next big change may come from better generalization or new learning rules, not from a better step downhill.

Teaching and learning with gradient descent

A physical model. Students can roll a marble on a bumpy tray and predict where it settles. Then point out the limit of the tray: it has only two directions, while a real model has millions, and in that many directions most flat spots are saddles with some way further down.

A tiny loop. This Python loop finds the minimum of (x - 3) squared. Students can change the learning rate to see slow progress, fast progress, and divergence.

x = 0.0
lr = 0.1
for step in range(50):
    grad = 2 * (x - 3)
    x = x - lr * grad
print(x)

A model for improvement. Gradient descent is a fair picture of deliberate practice: measure the error, make a small correction, repeat. Its limits as a metaphor are useful too. It needs a clear, measurable goal, it is greedy and can settle on a plateau, and it only improves what the loss function measures. People can change their goals and step back to rethink, which gradient descent cannot do. Learners comfortable with Calculus can go further with the free Deep Learning book.

Discussion questions, essay ideas, and learning related AI prompt ideas

• If gradient descent existed in 1847, what specifically had to change before it could train AlexNet in 2012, and which change mattered most?
• Should backpropagation and gradient descent be treated as one invention or two when ranking the most important ideas in AI?
• Why does a local, greedy method succeed on networks with billions of parameters, and what does the saddle point argument add to the explanation?
• Could a gradient-free method such as evolution strategies ever replace gradient descent for large language models, and what evidence would show it?
• What does Rich Sutton's "The Bitter Lesson" predict about the next decade of AI, and where might it be wrong?
• How well does gradient descent work as a model for human learning, and where does the comparison break down?
• Design a one-hour lesson that teaches gradient descent to high school students using only a marble, a tray, and a few lines of code.

See also

• Artificial intelligence
• Calculus
• Mathematics
• Computer science
• Statistics
• Open design GPU
• Running AI models locally
• Offline AI tutors for places without internet
• Learning