Model Tuning¶

Be data-driven with model tuning, by closely-examining actual performance

• Sometimes you need to decide if it is worth fixing certain type of error

Regularization¶

Methods that constrain the complexity of a model

• Goal: Improve out-of-sample error by reducing overfitting & variance
• Tradeoff: Compromising on an increased bias

Regularization allows a continuous spectrum of model complexity, rather than discrete jumps

Reducing Capacity¶

Reduce the number of

• parameters (weights)
• layers (neural net)
• units per layers (neural net)
• Bottle neck layers (neural net)

Bottleneck Layer(s) for neural nets¶

Let

• Let layer \(i\) and layer \(j\) be adjacent layers
• \(w_i\) be weights of layer \(i\)
• \(\vert w_i \vert\) be the number of units in a layer \(i\)
• \(w_b\) be the bottleneck layer
• effective only if \(\vert w_b \vert < \arg \min(\vert w_i \vert, \vert w_j \vert)\)

Example

flowchart LR
a[10<br/>units] -->|100<br/>connections| b[10<br/>units]
c[10<br/>units] -->|10<br/>connections| d[1<br/>unit] -->|10<br/>connections| e[10<br/>units]

Increase DOF¶

• Reduce \(k\)

• Feature Selection

• Dimensionality Reduction

• Increase \(n\)

• Data augmentation

Subsampling at each iteration¶

• columns
• rows

Weight Decay¶

Also called as

• Shrinkage Term
• Regularization Penalty

Reduce errors by fitting the function appropriately on the given training set, to help reduce variance, and hence avoid overfitting, while minimally affecting bias.

This is done by adding a penalty term in the cost function.

Note:

• All input features must be standardized
• Intercept should not be penalized
• You can use different penalty for each term
• You can perform different regularizer and norm based on expected knowledge of distribution of parameter

where

• \(\sigma_u =\) Random Standard Deviation: Stochastic Noise
• \(Q_f =\) Target Complexity: Deterministic Noise

This is similar to $$ \begin{aligned} E_\text{aug} &= E_\text{in} + \dfrac{\Omega}{n} \ &\updownarrow \ E_\text{out} &\le E_\text{in} + \dfrac{\Omega}{n} \end{aligned} $$

Hence, \(E_\text{aug}\) is a better proxy for \(E_\text{out}\) than \(E_\text{in}\)

where

• \(\mu_{\beta^*_j}\) is the prior-known most probable value of \(\beta_j\)
• \(\sigma_{\beta^*_j}\) is the prior-known standard deviation of \(\beta_j\)
• \(\gamma_j\): Weightage of weight decay
• Penalize some parameters more than others
• Useful to penalize higher order terms with \(\gamma_j = 2^q\), where \(q=\) complexity of term

\(\mu_{\beta^*_j}\) and \(\sigma_{\beta^*_j}\) incorporate desirable Bayesian aspects in our model.

Bayesian Interpretation¶

Regularization amounts to the use of informative priors, where we introduce our knowledge or belief about the target function in the form of priors, and use them to “regulate” the behavior of the hypothesis we choose

Incorporating \(\hat \beta\) into the regularization incorporates maximum likelihood estimate of the coefficients.

Didn’t understand: The standard deviation of the prior distribution corresponds to regularization strength \(\lambda\)

Example

IDK¶

Contours of Regularizers¶

Limitations¶

Magnitude of parameters may not always be the best estimate of complexity, especially for Deep Neural Networks

Why is standardization required?¶

• magnitudes of parameters need to be comparable
• Penalized estimates are not scale equivariant: multiplying \(x_j\) by a constant \(c\) can cause a significant change in \(\hat \beta\) when using regularization

Feature Selection Paths¶

Penalty Coefficient Magnitude¶

Penalty Coefficient vs Noise¶

Stochastic Noise	Deterministic Noise

where \(\lambda^* =\) Optimal \(\lambda\)

Frequentist Interpretation¶

\(\lambda\) regulates the smoothness of the spline that solves $$ \min_h \left { J(\theta, x_i, y_i) + \lambda \int [h''(t)]^2 \cdot dt \right } $$ Penalizing the squared \(k\)th derivative leads to a natural spline of degree \(2k − 1\)

IDK¶

Optimization Equivalent¶

This is equivalent to:

At each iteration, shrink the weights by the gradient of the regularization penalty before taking the gradient step

For L2: \((1-\nu \lambda)\) $$ \begin{aligned} w_{t+1} &= w_{t} - \nu \Big[ \nabla w_t + \nabla R(w_t) \Big] \ &= \Big[ 1 - \nu \nabla R(w_t) \Big] w_{t} - \nu \nabla w_t \

\implies \text{With L2} &= (1- \nu \lambda) w_{t} - \nu g(t) \end{aligned} $$

Most deep learning libraries incorporate weight decay in the optimizer; however, this isn’t the most conceptually best way to approach weight decay - better to incorporate in the loss function

Hinted Regularization¶

Penalize deviation from Model Hints

Multi-Stage Regularization¶

Intuition: Since vars in 2^nd stage have less "competition" from noise variables, estimating using selected variables could give better results

System Equation Penalty¶

Useful if you know the underlying systematic differential equation

$$ J'(\theta) = J(\theta) + \text{DE} \ \text{RHS(DE)} = 0 $$ Refer to PINNs for more information

IDK¶

Early Stopping¶

• This applies to all evaluation curves
• Stopping criteria
• In-Sample error < Out-Sample error
• Evaluation metric
  • RMSE \(\le \sigma_u\)
  • MAPE \(\le 1-\text{Accuracy}_\text{req}\)
• Stop few iterations/value before the optimal point, to reduce variance further: \(e_\text{stop} < e^*\)

Dropout¶

Dropout is applied on the output of hidden fully-connected layers

• Makes networks “robust” to missing activations
• Stochastic approximation

Training¶

Stochastically drop out units with probability \(p_\text{drop}\) and keep with \(p_\text{keep}=1-p_\text{drop}\)

Annealed dropout

Evaluation/Production¶

At inference time, dropout is inactive, as we should not predict stochastically

Ensembling¶

Reduces variance from high-variance models, such as trees

Noise Injection¶

• Add noise to inputs
• Add noise to outputs

Behaves similar to L2 regularization

Label Smoothing¶

The output of \(\sigma, \tanh\) never actually really output the maximum/minimum range values. So the model will keep trying to make the predictions go to exact \(0/1\) (which is never attainable), making it prone to overfitting

Using label smoothing, model becomes less confident with extremely confident labels (which we want to avoid). Now, the penalty given to a model due to an incorrect prediction will be slightly lower than using hard labels which would result in smaller gradients

where

• \(y_\text{true}\) is the true label
• \(C=\) total number of classes/labels
• \(\epsilon \approx 0\)

Extreme cases for \(\epsilon\)

• \(\epsilon=0\): original
• \(\epsilon=1\): uniform

Active Teaching¶

flowchart LR
c(( ))
m[ML]

d[(Dataset)] --> m & EDA

EDA -->
rb[Rule-Based System]

subgraph Active Teaching
  rb & m <-->|Compare & Update| c
end

Neural Network¶

Comments