Model Evaluation¶

Goals¶

1. Minimize bias
2. Minimize variance
3. Minimize generalization gap

Guidelines¶

• Always check if your model is able to learn from a synthetic dataset where you know the underlying data-generating process
• Metrics computed from test set may not be representative of the true population
• Always look at multiple metrics; never trust a single one alone
• false positives and false negatives are seldom equivalent
• understand the problem to known the right tradeoff
• Always monitor the worst-case prediction
• Maximum loss
• 95^th percentile loss

Baseline/Benchark models¶

Always establish a baseline

• Basic/Naive/Dummy predictions
• Regression
  • Mean
  • Max
  • Min
  • Random
• Classification
  • Mode
  • Random
• Time series specific
  • Persistence
  • \(\hat y_{t+h} = y_t\)
  • Latest value available
  • Great for short horizons
  • Climatology
  • \(\hat y_{t+h} = \bar y_{i \le t}\)
  • Average of all observations until present
  • Great for short horizons
  • Combination of Persistence and Climatology
  • \(\hat y_{t+h} = \beta_1 y_t + \beta_2 \bar y_{i \le t}\)
  • Lag: \(\hat y_{t+h} = y_{t-k}\)
  • Seasonal Lag: \(\hat y_{t+h} = y_{t+h-m}\)
  • Moving average
  • Exponential average
• Human Level Performance
• Literature Review
• Performance of older system

Significance¶

All the evaluation should be performed relative to the baseline.

For eg: Relative RMSE = RMSE(model)/RMSE(baseline), with “good” threshold as 1

Probabilistic Evaluation¶

Now, we need to see if any difference in accuracy across models/hyperparameters is statistically-significant, or just a matter of chance.

Summary Statistics: Don’t just look at the mean evaluation metric of the multiple splits; also get the uncertainty associated with the validation process.

• Standard error of accuracy estimate
• Standard deviation
• Quantiles
• PDF
• VaR

Bessel’s Correction¶

• where
• \(n=\) no of samples
• \(k=\) no of parameters
• \(e=\) no of intermediate estimates (such as \(\bar x\) for variance)
• Do not perform this on metrics which are already corrected for degree of freedom (such as \(R^2_\text{adj}\))
• Modify accordingly for squares/root metrics

Regression Evaluation¶

Probabilistic Evaluation¶

You can model the error such as MAE as a \(\chi^2\) distribution with dof = \(n-k\)

The uncertainty can be obtained from the distribution

Spurious Regression¶

Misleading statistical evidence of a relationship that does not truly exist

Occurs when we perform regression between

• 2 independent variables

and/or

• 2 non-stationary variables

(Refer econometrics)

You may get high \(R^2\) and \(t\) values, but \(u_t\) is not white noise (it is non-stationary)

\(\sigma^2_u\) becomes infinite as we go further in time

Classification Evaluation¶

There is always a tradeoff b/w specificity and sensitivity

Graphs¶

Graph			Preferred
Error Rate
ROC Curve		How does the classifier compare to a classifier that predicts randomly with \(p=\text{TPR}\) How well model can discriminate between \(y=0\) and \(y=1\)	Top-Left At least higher than 45deg line
Calibration Graph		Create bins of different predicted probabilities Calculate the fraction of \(y=1\) for each bin Confidence intervals (more uncertainty for bins with fewer samples) Histogram showing fraction of samples in each bin	Along 45deg line
Confusion Probabilities

Tradeoff for Threshold¶

Probabilistic Evaluation¶

Wilson score interval

You can model accuracy as a binomial distribution with

• \(n=\) Validation set size
• = No of predictions
• = No of k folds * Validation Set Size
• \(p=\) Obtained Accuracy of classifier

Similar to confidence intervals for proportion

The uncertainty can be obtained from the distribution

for n in [100, 1_000, 10_000, 100_000]:
  dist = stats.binom(n, 0.7)

  alpha = 0.025

  interval_width = dist.isf(alpha) - dist.isf(1-0.975)
  print(f"Size: {interval_width/n * 100}")
  # returns alpha % of observed accuracy that fall outside the inverval --> This is the maximum further accuracy that is theoretically achievable

Decision Boundary¶

Plot random distribution of values

For eg:

Confusion Matrix¶

\(n \times n\) matrix, where \(n\) is the number of classes

Binary Classification¶

Multi-Class Classification¶

Confusion Matrix with respect to A

Classification Accuracy Measures¶

Jacquard Index¶

Multi-Class Averaging¶

Inspection¶

Error Analysis¶

Residual Inspection

Perform all the inspection on

• train and dev data
• externally-studentized residuals, to correct for leverage

There should be no explainable unsystematic component

• Symmetric distribution for values of error terms for a given value \(x\)
• Not over time/different values of \(x\)
• This means that
• you have used up all the possible factors
• \(u_i\) only contains the non-systematic component

Why is this important?¶

For eg: Running OLS on Anscombe’s quartet gives

• same curve fit for all
• Same \(R^2\), RMSE, standard errors for coefficients for all

But clearly the fit is not equally optimal

1. Only the first one is acceptable
2. Model misspecification
3. Outlier
4. Leverage

which is shown in the residual plot

Aggregated Inspection¶

• Aggregate the data based on metadata
• Evaluate metrics on groups

where \(g()\) is the group, which can be \(x_{ij}, y_i\) or combination of these

• Image blurry/flipped/mislabelled
• Gender
• Age
• Age & Gender

Diebold-Mariano Test¶

Determine whether predictions of 2 models are significantly different

Basically the same as Comparing Samples for residuals

Bias-Variance Analysis¶

Evaluation Curves¶

Related to Interpretation

• Always look at all curves with uncertainties wrt each epoch, train, hyper-parameter value.
• The uncertainty in-sample and out-sample should also be similar
• If train set metric uncertainty is small and test set metric uncertainty is large, this is bad even if the average loss metric is same

Learning Curve¶

Based on the slope of the curves, determine if adding more data will help

	Conclusion
	High Bias (Underfitting)
	High Variance (Overfitting)
	High Bias High Variance

Loss Curve¶

Same Model, Variable Learning Rate¶

Validation Curve¶

Neural Network¶

Comments