Model Evaluation¶

Goals¶

1. Minimize bias
2. Minimize variance
3. Minimize generalization gap

Guidelines¶

• Always check if your model is able to learn from a synthetic dataset where you know the underlying data-generating process
• Always evaluate model with different random seeds to ensure accurate result
  • Group by random seed and check for discrepancies
• Metrics computed from test set may not be representative of the true population
• Always look at multiple metrics; never trust a single one alone
• false positives and false negatives are seldom equivalent
• understand the problem to known the right tradeoff
• Always monitor the worst-case prediction
• Maximum loss
• 95^th percentile loss

Baseline/Benchark models¶

Always establish a baseline

• Basic/Naive/Dummy predictions
• Regression
  • Mean
  • Max
  • Min
  • Random
• Classification
  • Mode
  • Random
• Time series specific
  • Persistence
  • \(\hat y_{t+h} = y_t\)
  • Latest value available
  • Great for short horizons
  • Climatology
  • \(\hat y_{t+h} = \bar y_{i \le t}\)
  • Average of all observations until present
  • Great for short horizons
  • Combination of Persistence and Climatology
  • \(\hat y_{t+h} = \beta_1 y_t + \beta_2 \bar y_{i \le t}\)
  • Lag: \(\hat y_{t+h} = y_{t-k}\)
  • Seasonal Lag: \(\hat y_{t+h} = y_{t+h-m}\)
  • Moving average
  • Exponential average
• Human Level Performance
• Literature Review
• Performance of older system

Significance¶

All the evaluation should be performed relative to the baseline.

For eg: Relative RMSE = RMSE(model)/RMSE(baseline), with “good” threshold as 1

Probabilistic Evaluation¶

Now, we need to see if any difference in accuracy across models/hyperparameters is statistically-significant, or just a matter of chance - Try different seeds and see the range; this range is purely due to randomness

Summary Statistics: Don’t just look at the mean evaluation metric of the multiple splits; also get the uncertainty associated with the validation process.

It is invalid to use a standard hypothesis test - such as a T-test - across the different folds of a cross-validation, as the runs are not independent. - It is acceptable for a fixed test set - Std obtained in model performance from cross-validation is standard error of mean estimate - Std obtained in model performance from cross-validation is standard error of mean estimate - True standard deviation is no of folds x SE

Hence, treat the metrics across the folds as a a distribution - For regression: chi squared - For classification: binomial

• Standard error of accuracy estimate
• Standard deviation
• Quantiles
• PDF
• VaR

Bessel’s Correction¶

• where
• \(n=\) no of samples
• \(k=\) no of parameters
• \(e=\) no of intermediate estimates (such as \(\bar x\) for variance)
• Do not perform this on metrics which are already corrected for degree of freedom (such as \(R^2_\text{adj}\))
• Modify accordingly for squares/root metrics

Regression Evaluation¶

Metric	Formula	Unit	Range	Preferred Value	Signifies	Advantages ✅	Disadvantages ❌	Comment
\(R^2\) (Coefficient of Determination)	\(1 - \text{RSE}\)	Unitless	\([0, 1]\)	\(1\)	Proportion of changes in dependent var explained by regressors. Proportion of variance in \(y\) explained by model wrt variance explained by mean Demonstrates ___ of regressors - Relevance - Power - Importance		Cannot use to compare same model on different samples, as it depends on variance of sample Susceptible to spurious regression, as it increases automatically when increasing predictors
Adjusted \(R^2\)	\(1 - \left[\dfrac{(n-1)}{(n-k-1)} (1-R^2)\right]\)	Unitless	\([0, 1]\)	\(1\)		Penalizes large number of predictors
Accuracy	\(100 - \text{MAPE}\)	%	\([0, 100]\)	\(100 \%\)
\(\chi^2_\text{reduced}\)	\(\dfrac{\chi^2}{n-k} = \dfrac{1}{n-k}\sum \left( u_i/\sigma_i \right)^2\)		\([0, \infty]\)	\(1\)				\(\approx 1:\) Good fit \(\gg 1:\) Underfit/Low variance estimate \(\ll 1:\) Overfit/High variance estimate
Spearman’s Correlation	\(\dfrac{ r(\ rg( \hat y), rg(y) \ ) }{ \sigma(\ rg(\hat y) \ ) \cdot \sigma( \ rg(y) \ ) }\)	Unitless	\([-1, 1]\)	\(1\)		Very robust against outliers Invariant under monotone transformations of \(y\)
DW (Durbin-Watson Stat)				\(> 2\)	Confidence of error term being random process		Not appropriate when \(k>n\)	Similar to \(t\) or \(z\) value If \(R^2 >\) DW Statistic, there is Spurious Regression
AIC Akaike Information Criterion	\(-2 \ln L + 2k\)			\(0\)	Leave-one-out cross validation score	Penalizes predictors more heavily than \(R_\text{adj}^2\)	For small values of \(n\), selects too many predictors Not appropriate when \(k>n\)
AIC Corrected	\(\text{AIC} + \dfrac{2k(k+1)}{n-k-1}\)			\(0\)		Encourages feature selection
BIC/SBIC/SC (Schwarz’s Bayesian Information Criterion)	\(-2 \ln L + k \ln n\)			\(0\)		Penalizes predictors more heavily than AIC
HQIC Hannan-Quinn Information Criterion	\(-2 \ln L + 2k \ln \vert \ln n \vert\)			\(0\)
Calibration
Sharpness

Probabilistic Evaluation¶

You can model the error such as MAE as a \(\chi^2\) distribution with dof = \(n-k\)

The uncertainty can be obtained from the distribution

Spurious Regression¶

Misleading statistical evidence of a relationship that does not truly exist

Occurs when we perform regression between

• 2 independent variables

and/or

• 2 non-stationary variables

(Refer econometrics)

You may get high \(R^2\) and \(t\) values, but \(u_t\) is not white noise (it is non-stationary)

\(\sigma^2_u\) becomes infinite as we go further in time

Classification Evaluation¶

There is always a tradeoff b/w specificity and sensitivity

Graphs¶

Graph			Preferred
Error Rate
ROC Curve		How does the classifier compare to a classifier that predicts randomly with \(p=\text{TPR}\) How well model can discriminate between \(y=0\) and \(y=1\)	Top-Left At least higher than 45deg line
Calibration Graph		Create bins of different predicted probabilities Calculate the fraction of \(y=1\) for each bin Confidence intervals (more uncertainty for bins with fewer samples) Histogram showing fraction of samples in each bin	Along 45deg line
Confusion Probabilities

Tradeoff for Threshold¶

Probabilistic Evaluation¶

For simple metrics that rely on counting successes, such as accuracy, sensitivity, PPV, NPV - the sampling distribution can be deduced from a binomial law - uncertainty via Proportion Testing

• \(n=\) Validation set size
  • = \(\sum \limits_i^k n_\text{fold}\)
• \(\hat p =\) Average Accuracy of classifier = Estimated proportion
  • \(= \dfrac{1}{k} \sum \limits_i^k \text{Accuracy}_\text{fold}\)

Decision Boundary¶

Plot random distribution of values

For eg:

Confusion Matrix¶

\(n \times n\) matrix, where \(n\) is the number of classes

Binary Classification¶

Multi-Class Classification¶

Confusion Matrix with respect to A

Classification Accuracy Measures¶

Jacquard Index¶

Multi-Class Averaging¶

Inspection¶

Error Analysis¶

Residual Inspection

Perform all the inspection on

• train and dev data
• externally-studentized residuals, to correct for leverage

There should be no explainable unsystematic component

• Symmetric distribution for values of error terms for a given value \(x\)
• Not over time/different values of \(x\)
• This means that
• you have used up all the possible factors
• \(u_i\) only contains the non-systematic component

Why is this important?¶

For eg: Running OLS on Anscombe’s quartet gives

• same curve fit for all
• Same \(R^2\), RMSE, standard errors for coefficients for all

But clearly the fit is not equally optimal

1. Only the first one is acceptable
2. Model misspecification
3. Outlier
4. Leverage

which is shown in the residual plot

Aggregated Inspection¶

• Aggregate the data based on metadata
• Evaluate metrics on groups

where \(g()\) is the group, which can be \(x_{ij}, y_i\) or combination of these

• Image blurry/flipped/mislabelled
• Gender
• Age
• Age & Gender

Diebold-Mariano Test¶

Determine whether predictions of 2 models are significantly different

Basically the same as Comparing Samples for residuals

Bias-Variance Analysis¶

Evaluation Curves¶

Related to Interpretation

• Always look at all curves with uncertainties wrt each epoch, train, hyper-parameter value.
• The uncertainty in-sample and out-sample should also be similar
• If train set metric uncertainty is small and test set metric uncertainty is large, this is bad even if the average loss metric is same

Learning Curve¶

Based on the slope of the curves, determine if adding more data will help

	Conclusion
	High Bias (Underfitting)
	High Variance (Overfitting)
	High Bias High Variance

Loss Curve¶

Same Model, Variable Learning Rate¶

Validation Curve¶

Neural Network Distributions¶

Avoids

• Saturation
• Exploding

Example Plot¶

Object Detection¶

IDK¶

Comments