Activation Functions
AI · Intermediate
Definition
Activation functions add non-linearity to neurons so networks can learn complex patterns.
Without them a deep network collapses into a single linear model.
Diagram
Flowchart
Example: ReLU and sigmoid
import math
z = -2
print(max(0, z), round(1 / (1 + math.exp(-z)), 3))Output: 0 0.119
Key points
- ReLU is default for hidden layers
- Softmax for multi-class outputs
- Sigmoid saturates → vanishing gradients
Interview questions
Why ReLU over sigmoid?
It does not saturate for positive inputs, so gradients flow better.