Predictive model

In this lecture we introduce the concept of a predictive model, which lies at the heart of machine learning (ML).

Table of contents

Observed input-output mapping
Prediction
Some machine learning jargon
Loss function
Risk
Empirical risk
Empirical risk minimization

Observed input-output mapping

To begin with, we observe some outputs and the corresponding input vectors that may help to predict the outputs before they are observed.

Examples:

$y_{t}$ is the total amount of purchases made by a customer while visiting an online shop; $x_{t}$ are some characteristics of the landing page that was first seen by the customer;
$y_{t}$ is inflation observed in month and $x_{t}$ is a vector of macro-economic variables known before ;
$y_{t}$ is 1 if firm defaults within a year and 0 otherwise; $x_{t}$ is a vector of firm 's characteristics that may help to predict the default;
$y_{t}$ is a measure of economic activity in province ; $x_{t}$ is a vector of pixel values from a satellite image of the province.

Note: the subscript used to index the observations is not necessarily time.

Prediction

We use the observed inputs and outputs to build a predictive model, that is, a function that takes new inputs as arguments and returns the predictions [eq4] of previously unseen outputs

Some machine learning jargon

Before diving into predictive modelling, let us learn some machine learning jargon.

Supervised vs unsupervised learning

The problem of learning an input-output mapping is called a supervised learning problem.

The data used for learning is called labelled data and the outputs $y_{t}$ are called labels or targets.

Basically, in a supervised learning problem, the task is to learn the conditional distribution of the outputs given the inputs.

On the contrary, in an unsupervised learning problem, there are no labels and the task is to learn something about the unconditional distribution of $x_{t}$ .

The typical example are photos of cats and dogs: $x_{t}$ is a vector of pixel values; in supervised learning, you have labels $y_{t}$ (1 if dog, 0 if cat); in unsupervised learning, you have no labels, but you typically do something like clustering $x_{t}$ in the hope that the algorithm autonomously separates cats from dogs.

Regression vs classification

A supervised learning problem is called:

a classification problem if the output variable $y_{t}$ is discrete / categorical (e.g., cat vs dog);
a regression problem if the output variable $y_{t ext{ }}$ is continuous (e.g., income earned).

Features

The inputs are often called features and the vector $x_{t}$ is called a feature vector.

Training

The act of using data to find the best predictive model (e.g., by optimizing the parameters of a parametric model) is called model training.

Loss function

How do we assess the quality of a predictive model?

How do we compare predicted outputs $widetilde{y}_t$ with observed outputs ?

We do these things by specifying a loss function, which is always required in a machine learning problem.

A loss function quantifies the losses that we incur when we make inaccurate predictions.

Examples:

Squared Error (SE):
Absolute Error (AE):
Log-loss (or cross-entropy): when $y_{t}$ is binary (i.e., it can take only two values, either 0 or 1); the multivariate generalization is when $y_{t}$ is a multinoulli vector (i.e., we have a categorical variable that can take only values; when it takes the -th, then $y_{t,k}=1$ and all the other entries of the vector $y_{t}$ are zero).

Risk

Ideally, the best predictive models is the one having the smallest statistical risk (or expected loss) where the expected value is with respect to the joint distribution of $x_{t}$ and $y_{t}$ .

Empirical risk

Since the true joint distribution of $x_{t}$ and $y_{t}$ is usually unknown, the risk is approximated by the empirical risk [eq11] where is a set of input-output pairs used for calculating the empirical risk and is its cardinality (the number of input-output pairs contained in ).

Thus, the empirical risk is the sample average of the losses over a set of observed data . This is the reason why we sometimes call it average loss.

How to choose is one of the most important decisions in machine learning and we will discuss it at length.

For specific choices of the loss function, empirical risk has names that are well-known to statisticians:

if the loss is the Squared Error, then the empirical risk is the Mean Squared Error (MSE), and its square root is the Root Mean Squared Error (RMSE);
if the loss is the Absolute Error, then the empirical risk is the Mean Absolute Error (MAE);
if the loss is the Cross-Entropy, it can easily be proved that the empirical risk is equal to the negative average log-likelihood.

Empirical risk minimization

The criterion generally followed in machine learning is that of empirical risk minimization:

if we are setting the parameters of a model, we choose the parameters that minimize the empirical risk;
if we are choosing the best model in a set of models, we pick the one that has the lowest empirical risk.

Statistically speaking, it is a sound criterion because empirical risk minimizers are extremum estimators.

How to cite

Please cite as:

Taboga, Marco (2021). "Predictive model", Lectures on machine learning. https://www.statlect.com/machine-learning/predictive-model.