02 Data

Data can be anything. It depends on the data engineer on what the input and output data is.

Types¶

• Structured
• Numbers
• Tables
• Unstructured
• Audio
• Image
• Video

Means of data collection¶

Garbage-in, Garbage-out

• Manual Labelling
• Manually marking as cat/not cat, etc.
• Observing Behaviour
• taking data from user activity and seeing whether they purchased or not
• machine temperatures and observing for faults or not
• Download from the web

Mistakes¶

1. Waiting too long for implementing a data set
2. implement it early so that AI team can give feedback to the IT team
3. Not all data is valuable
4. Messy
5. Garbage in, garbage out
6. incorrect data
7. multiple types of data

Datasets¶

Collection of data in rows and columns

• Rows = Objects, Records, Samples, Instances
• Columns = Attributes, Variables, Dimensions, Features

Types¶

• Labelled has Target variable
• Unlabelled does not have target variable

Types of Attributes¶

• D = Discrete/Qualitative/Categorical
• N = Numerical/Quantitative/Continuous

Asymmetric Attributes¶

Attributes where only non-zero values are important. It can be

• Binary (0 or 1)
• Discrete (0, 1, 2, 3, …)
• Continuous (0, 33.35, 52.99, …)

Characteristics of Dataset¶

Sample Size¶

where

• \(n =\) no of sample points
• \(k =\) no of input variables
• \(C =\) no of classes

Dimensionality¶

No of features

Sparseness¶

If majority of attributes have 0 as value, depending on the context

Resolution¶

Detail/Frequency of the data (hourly, daily, monthly, etc)

Types of Datasets¶

Records¶

Collection of records having fixed attributes, without any relationship with other records

Graph¶

Ordered¶

Relationships between attributes

Sequential/Temporal¶

Extension of record, where each record has a time associated with it.

Even this can be time series data, if recorded periodically.

Time-Associated¶

Sequence Data¶

Sequence of entities

Eg: Genomic sequence data

Time Series Data¶

Series of observations over time recorded periodically

Each record is a time series as well.

Spatial Data¶

Data has spatial attributes, such as positions/areas

Weather data collected for various locations

Spatio-Temporal Data¶

Data has both spatial and temporal attributes

Issues with Data Quality¶

Estimation¶

Data¶

Data can be structured/unstructured

• Each column = feature
• Each row = instance

Data Split¶

• Train-Inner Validation-Outer Validation-Test is usually 60:10:10:20
• Split should be mutually-exclusive, to ensure good out-of-sample accuracy

The size of test set is important; small test set implies statistical uncertainty around the estimated average test error, and hence cannot claim algo A is better than algo B for given task.

Random split is the best. However, random split will not work well all the time, where there is auto-correlation, for eg: time-series data

flowchart LR

td[(Training Data)] -->
|Training| m[Model] -->
|Validation| vd[(Validation)] -->
|Tuning| m --->
|Testing| testing[(Testing Data)]

Multi-Dimensional Data¶

can be hard to work with as

• requires more computing power
• harder to interpret
• harder to visualize

Feature Selection¶

Dimension Reduction¶

Using Principal Component Analysis

Deriving simplified features from existing features

Easy example: using area instead of length and breadth.

Categories of Data¶

“Fat-Tailedness”¶

Degree to which rare events drive the aggregate statistics of a distribution

• Lower \(\alpha \implies\) Fatter tails
• 
• Kurtosis (breaks down for \(\alpha \le 4\))
• Variance of Log-Normal distribution
• 
• Taleb’s \(\kappa\) metric
• 

Comments