Gradient Descent
AI · Intermediate
Definition
Gradient descent minimises a loss by repeatedly moving parameters a small step opposite to the gradient.
It is how almost every ML and deep learning model is trained.
Diagram
Flowchart
Example: Minimise (w−3)²
w, lr = 0.0, 0.1
for _ in range(50):
w -= lr * 2 * (w - 3)
print(round(w, 3))Output: 3.0
Key points
- Learning rate controls step size
- SGD / mini-batch / Adam variants
- Can get stuck in local minima
Interview questions
What if the learning rate is too high?
Updates overshoot and the loss may diverge.