Gradient Boosting & XGBoost
Machine Learning · Advanced
Definition
Gradient boosting builds trees one after another, each fixing the errors of the previous ones.
Top performer on tabular data and Kaggle competitions.
Diagram
Flowchart
Example: Residual update
y, pred, lr = 10, 6, 0.5
for _ in range(3):
pred += lr * (y - pred)
print(pred)Output: 9.5
Key points
- Lower learning rate + more trees = better generalization
- Use early stopping
Interview questions
Bagging vs boosting?
Bagging trains in parallel to reduce variance; boosting trains sequentially to reduce bias.