Search for probability and statistics terms on Statlect
StatLect

Point estimation

by , PhD

Point estimation is a type of statistical inference which consists in producing a guess or approximation of an unknown parameter.

In this lecture we introduce the theoretical framework that underlies all point estimation problems.

At the end of the lecture, we provide links to detailed examples of point estimation, in which we show how to apply the theory.

Table of Contents

Sample and data-generating distribution

The main elements of a point estimation problem are those found in any statistical inference problem:

Parametric model

When the model $Phi $ is put into correspondence with a set [eq2] of real vectors, then we have a parametric model.

The set $Theta $ is called the parameter space and its elements are called parameters.

Denote by $	heta _{0}$ the parameter that is associated with the data-generating distribution [eq3] and assume that $	heta _{0}$ is unique. The vector $	heta _{0}$ is called the true parameter.

Estimate and estimator

Point estimation is the act of choosing a vector [eq4] that approximates $	heta _{0}$. The approximation $widehat{	heta }$ is called an estimate (or point estimate) of $	heta _{0}$.

When the estimate $widehat{	heta }$ is produced using a predefined rule (a function) that associates a parameter estimate $widehat{	heta }$ to each $xi $ in the support of $Xi $, we can write[eq5]

The function [eq6] is called an estimator.

Often, the symbol $widehat{	heta }$ is used to denote both the estimate and the estimator. The meaning is usually clear from the context.

Estimation error

According to the decision-theoretic terminology introduced previously, making an estimate $widehat{	heta }$ is an act, which produces consequences.

Among these consequences, the most relevant one is the estimation error[eq7]

The statistician's goal is to commit the smallest possible estimation error.

Loss

The preference for small errors can be formalized with a loss function [eq8]that quantifies the loss incurred by estimating $	heta _{0}$ with $widehat{	heta }$.

Examples of loss functions are:

  1. the absolute error:[eq9]where [eq10] is the Euclidean norm (it coincides with the absolute value when [eq11]);

  2. the squared error:[eq12]

Risk

When the estimate $widehat{	heta }$ is obtained from an estimator, it is a function of the random vector $Xi $ and the loss [eq13]is a random variable.

The expected value of the loss[eq14]is called the statistical risk (or, simply, the risk) of the estimator $widehat{	heta }$.

Estimates of risk

The expected value in the definition of risk is computed with respect to the true distribution function [eq15].

Therefore, we can compute the risk [eq16] only if we know the true parameter $	heta _{0}$ and [eq17].

When $	heta _{0}$ and [eq17] are unknown, the risk needs to be estimated.

For example, we can approximate the risk with the quantity [eq19]where:

Even if the risk is unknown, the notion of risk is often used to derive theoretical properties of estimators.

Risk minimization

Point estimation is always guided, at least ideally, by the principle of risk minimization, that is, by the search for estimators that minimize the risk.

Common risk measures

Depending on the specific loss function we use, the statistical risk of an estimator can take different names:

  1. when the absolute error is used as a loss function, then the risk[eq22]is called the Mean Absolute Error (MAE) of the estimator.

  2. when the squared error is used as a loss function, then the risk[eq23]is called Mean Squared Error (MSE). The square root of the mean squared error is called root mean squared error (RMSE).

Other criteria to evaluate estimators

In this section we discuss other criteria that are commonly used to evaluate estimators.

Unbiasedness

If an estimator produces parameter estimates that are on average correct, then it is said to be unbiased.

The following is a formal definition.

Definition Let $	heta _{0}$ be the true parameter. An estimator $widehat{	heta }$ is an unbiased estimator of $	heta _{0}$ if and only if[eq24]If an estimator is not unbiased, then it is called a biased estimator.

If an estimator is unbiased, then the estimation error is on average zero:[eq25]

Consistency

If an estimator produces parameter estimates that converge to the true value when the sample size increases, then it is said to be consistent.

The following is a formal definition.

Definition Let [eq26] be a sequence of samples such that all the distribution functions [eq27] are put into correspondence with the same parameter $	heta _{0}$. A sequence of estimators [eq28] is said to be consistent (or weakly consistent) if and only if[eq29]where $QTR{rm}{plim}$ indicates convergence in probability. The sequence of estimators is said to be strongly consistent if and only if[eq30]where [eq31] indicates almost sure convergence. A sequence of estimators which is not consistent is called inconsistent.

When the sequence of estimators is obtained using the same predefined rule for every sample $xi _{n}$, we often say, with a slight abuse of language, "consistent estimator" instead of saying "consistent sequence of estimators". In such cases, what we mean is that the predefined rule produces a consistent sequence of estimators.

Examples

You can find detailed examples of point estimation in the lectures on:

How to find a point estimator

The methods to find point estimators are called estimation methods.

You can read about these methods here:

Point vs interval estimation

There is another kind of estimation, called set estimation or interval estimation.

While in point estimation we produce a single estimate meant to approximate the true parameter, in set estimation we produce a whole set of estimates meant to include the true parameter with high probability.

How to cite

Please cite as:

Taboga, Marco (2021). "Point estimation", Lectures on probability theory and mathematical statistics. Kindle Direct Publishing. Online appendix. https://www.statlect.com/fundamentals-of-statistics/point-estimation.

The books

Most of the learning materials found on this website are now available in a traditional textbook format.