Optimizers (SGD, Momentum, Adam)
AI · Advanced
Definition
Optimizers decide how weights are updated from gradients; Adam adapts the step size per parameter.
Good optimizers train faster and more stably.
Diagram
Flowchart
Example: Momentum steps
w, v, lr, beta = 5.0, 0.0, 0.1, 0.9
for _ in range(2):
g = 2 * w
v = beta * v + g
w -= lr * v
print(round(w, 3))Output: 2.3
Key points
- Adam is a strong default
- Learning-rate schedules help convergence
Interview questions
What does momentum fix?
It smooths noisy gradients and speeds movement along consistent directions.