ANN - Model/Weight/Parameter Regularization/Regularizer Methods/Techniques
General ML Regularization Techniques
- Early Stopping
- L1/L2 Regularization
- Max Norm Constraints/Regularization
- Regularization - Parameter Weight Decay
ANN Specific Regularization Techniques
Regularization Methods Comparisons on MNIST
| Method | Test Classification Error % |
|---|---|
| L2 | 1.62 |
| L2 + L1 applied towards the end of training | 1.60 |
| L2 + KL-sparsity | 1.55 |
| Max-Norm | 1.35 |
| Dropout + L2 | 1.25 |
| Dropout + Max-Norm | 1.05 |
From Dropout: A Simple Way to Prevent Neural Networks from Overfitting
, multiple selections available,