Model¶

where

IDK¶

flowchart LR
d[Data<br/>Generation] -->
|Input| m[Modelling] -->
|Analysis| si[Scientific<br/>Investigation]

si -->
|Improve Model| m

si -->
|Improve DoE/Data Generation| d

Desired Properties¶

• Unbiased: Mean of residuals = 0
• Efficient: Variance of residuals and learnt parameters is min
• Maximum likelihood \(P(D, \theta)\)
• Robust
• Consistent: \(n \to \infty \implies E[u_i] \to 0\)

Attributes of probabilistic forecast quality

1. Reliable: probabilistic calibration
2. For quantile forecasts with level \(\alpha\), observations \(y_{t+k}\) should be less than \(\hat y_{t+k}\) \(\alpha\) times
3. For interval forecasts with coverage \(p\), observations \(y_{t+k}\) should be within the interval \(p\) times
4. For predictive densities composed of \(m+1\) quantile forecasts with nominal levels \(\alpha_0, \alpha_1, \dots, \alpha_m\), all these quantile forecasts are evaluated individually using the above
5. Q-Q Plots
6. Sharp: informative
7. Concentration of probability: how tight the predictive densities are
8. Perfect probabilistic forecast gives a probability of 100% on a single value
9. CRPS
   1. Average of each predictive density and corresponding observation
   2. \(\text{CRPS}_{t, h} = \int_y \ \Big( \hat F_{t+h \vert t} - 1(y_{t+h} \le y) \Big)^2 \ \cdot dy\)
   3. \(\text{CRPS}_h = \text{avg}(\text{CRPS}_{t, h})\)
10. Skilled
11. High resolution

Note¶

Every model is only limited to its ‘scope’, which should be clearly documented

IDK¶

• "If you understand your solution better than the problem, then you are doing something wrong" ~ Vincent Warmerdam

• Think more about system design rather than just machine learning
• Simple linear models work. Most of the times non-linear/ensembles/deep learning models are not required

Model Types¶

	Ideal	Non-Parametric (Nearest Neighbor)	Semi-Parametric	Parametric
\(\hat y\)	\(\text{Mean}(y \vert x)\)	\(\text{Mean} \Big(y \vert x_i \in N(x) \Big)\) \(N(x)\) is neighborhood of \(x\)		\(f(x)\)
Functional Form assumption	None	None		Assumes functional form with a finite & fixed number of parameters, before data is observed
Advantages	Perfect accuracy	- learns complex patterns - in a high-dimensional space - without being specifically directed - learns interactions		Compression of model into a single function
Limitation	Not possible to obtain	Suffers from curse of dimensionality: Requires large dataset, especially when \(k\) is large Black box: Lacks interpretability Large storage cost: Stores all training records Computationally-expensive		Lost information?
Visualization
Space Complexity is function of		Training set size	Number of function parameters	Number of function parameters
Example		Nearest Neighbor averaging	Spline	Linear Regression

Fundamentally, a parametric model can be though of data compression

Modelling Types¶

Structural vs Reduced-Form¶

Number of Variables¶

Degree of Model¶

The term linear refers to the linearity in the coefficients \(\beta\)s, not the predictors

Jensen’s Inequality¶

Therefore $$ \hat y = \exp(\beta_0 + \beta_1 x) + u_i \ E[y \vert x] \ne E[\exp(\beta_0 + \beta_1 x)] $$ However, if you assume that \(u \sim N(0, \sigma^2)\) $$ E[y \vert x] = \exp(\beta_0 + \beta_1 x + \dfrac{\sigma^2}{2}) $$

Multi-Layer Models¶

Boosting with different function and/or model class for each component of \(f(x)\) $$ \hat f(x) = \sum_{j=1}^k \hat f_j(x_j) \ \hat f: x_j \to u_{j-1} \ u_0 = y $$

flowchart LR
x1 --> h1
y ---> u1
h1 --> u1

x2 --> h2
u1 --> u2
h2 ---> u2

Model in the following order to avoid fitting the noise:

1. Components: low-frequency components first, then high-frequency components
2. Model: low-variance models preferred, then high-variance models; or you can optimize the entire thing as a carefully-crafted neural network

For eg: Time Series modelling

• Trend with LR: \(\hat f(t)\)
• Seasonality with KNN
• Holidays with Decision Tree

Categorial Inputs¶

Binary¶

where \(T=\) treatment/binary var

Discrete Var¶

For \(C\) possible values of discrete var, you need to create \((C-1)\) dummy vars, as all zeros is also scenario

Else, if you have \(C\) dummy vars, you will have perfect multi-collinearity (dangerous)

Model Hints¶

Known properties of \(f\) that can be used to improve \(\hat f\), especially with small datasets

• Monotonicity
• Reflexivity
• Symmetry
• Rotational invariance
• Translational invariance

Can be enforced through

• Modifying features
• Eg: using \(\vert x \vert\) instead of \(x\) for symmetry
• Regularization Penalty
• Data augmentation

Latent Variable Models¶

Examples¶

1. Image classification

2. Contains variability due to gender, eye color, hair color, pose, etc

3. Unless these images are annotated, these factors of variation are not explicitly available

4. Classification

5. Gaussian mixture models

Limitations¶

• Computationally-expensive: requires approximations

Comments