Handling Imbalanced Data
Machine Learning · Intermediate
Definition
Imbalanced data has far fewer examples of one class, so models may ignore it.
Fraud, disease and churn datasets are almost always imbalanced.
Diagram
Flowchart
Example: Class weights
counts = {'normal': 990, 'fraud': 10}
n = sum(counts.values())
print({k: round(n / (2 * v), 2) for k, v in counts.items()})Output: {'normal': 0.51, 'fraud': 50.0}
Key points
- Accuracy is misleading here
- Resample only the training set
Interview questions
Why is 99% accuracy bad here?
Predicting 'normal' always gets 99% while catching zero fraud.