StatlectThe Digital Textbook
Index > Fundamentals of statistics > Point estimation

Point estimation of the variance

This lecture presents some examples of point estimation problems, focusing on variance estimation, i.e. on using a sample to produce a point estimate of the variance of an unknown distribution.

Normal IID samples - Known mean

In this example we make assumptions that are similar to those we made in the example of mean estimation entitled Mean estimation - Normal IID samples. The reader is strongly advised to read that example before reading this one.

The sample

The sample $xi _{n}$ is made of n independent draws from a normal distribution having known mean mu and unknown variance sigma^2. Specifically, we observe n realizations $x_{1}$, ..., $x_{n}$ of n independent random variables X_1, ..., X_n, all having a normal distribution with known mean mu and unknown variance sigma^2. The sample is the n-dimensional vector [eq1] which is a realization of the random vector [eq2]

The estimator

We use the following estimator of variance:[eq3]

Expected value of the estimator

The expected value of the estimator [eq4] is equal to the true variance sigma^2:[eq5]

Proof

This can be proved using linearity of the expected value:[eq6]

Therefore, the estimator [eq4] is unbiased.

Variance of the estimator

The variance of the estimator [eq4] is:[eq9]

Proof

This can be proved using the fact that for a normal distribution [eq10] and the formula for the variance of an independent sum:[eq11]

Therefore, the variance of the estimator tends to zero as the sample size n tends to infinity.

Distribution of the estimator

The estimator [eq4] has a Gamma distribution with parameters n and sigma^2.

Proof

The estimator [eq4] can be written as:[eq14]where the variables $Z_{i}$ are independent standard normal random variables and $W$, being a sum of squares of n independent standard normal random variables, has a Chi-square distribution with n degrees of freedom (see the lecture entitled Chi-square distribution for more details). Multiplying a Chi-square random variable with n degrees of freedom by $sigma ^{2}/n$ one obtains a Gamma random variable with parameters n and sigma^2 (see the lecture entitled Gamma distribution for more details).

Risk of the estimator

The mean squared error of the estimator is:[eq15]

Consistency of the estimator

The estimator[eq3]can be viewed as the sample mean of a sequence [eq17] where the generic term of the sequence is[eq18]The sequence [eq17] satisfies the conditions of Kolmogorov's Strong Law of Large Numbers ([eq20] is an IID sequence with finite mean). Therefore, the sample mean of $Y_{n}$ converges almost surely to the true mean [eq21]:[eq22]Therefore the estimator [eq23] is strongly consistent. It is also weakly consistent, because almost sure convergence implies convergence in probability:[eq24]

Normal IID samples - Unknown mean

This example is similar to the previous one. The only difference is that we relax the assumption that the mean of the distribution is known.

The sample

The sample $xi _{n}$ is made of n independent draws from a normal distribution having unknown mean mu and unknown variance sigma^2. Specifically, we observe n realizations $x_{1}$, ..., $x_{n}$ of n independent random variables X_1, ..., X_n, all having a normal distribution with unknown mean mu and unknown variance sigma^2. The sample is the n-dimensional vector [eq25] which is a realization of the random vector [eq2]

The estimator

In this example also the mean of the distribution, being unknown, needs to be estimated. It is estimated with the sample mean Xbar_n:[eq27]

We use the following estimators of variance:

  1. the unadjusted sample variance:[eq28]

  2. the adjusted sample variance:[eq29]

Expected value of the estimator

The expected value of the unadjusted sample variance is:[eq30]

Proof

This can be proved as follows:[eq31]But [eq32] when $i
eq j$ (because X_i and $X_{j}$ are independent when $i
eq j$ - see Mutual independence via expectations). Therefore:[eq33]

Therefore, the unadjusted sample variance $S_{n}^{2}$ is a biased estimator of the true variance sigma^2. The adjusted sample variance $s_{n}^{2}$, on the contrary, is an unbiased estimator of variance:[eq34]

Proof

This can be proved as follows:[eq35]

Thus, when also the mean mu is being estimated, we need to divide by $n-1 $ rather than by n to obtain an unbiased estimator. Intuitively, by considering squared deviations from the sample mean rather than squared deviations from the true mean, we are underestimating the true variability of the data. In fact, the sum of squared deviations from the true mean is always larger than the sum of squared deviations from the sample mean. Dividing by $n-1$ rather than by n exactly corrects this bias. The number $n-1$ by which we divide is called the number of degrees of freedom and it is equal to the number of sample points (n) minus the number of other parameters to be estimated (in our case 1, the true mean mu).

The factor by which we need to multiply the biased estimatot $S_{n}^{2}$ to obtain the unbiased estimator $s_{n}^{2}$ is, of course:[eq36]

This factor is known as degrees of freedom adjustment, which explains why $S_{n}^{2}$ is called unadjusted sample variance and $s_{n}^{2}$ is called adjusted sample variance.

Variance of the estimator

The variance of the unadjusted sample variance is:[eq37]

Proof

This is proved in the following subsection (distribution of the estimator).

The variance of the adjusted sample variance is:[eq38]

Proof

This is also proved in the following subsection (distribution of the estimator).

Therefore, both the variance of $S_{n}^{2}$ and the variance of $s_{n}^{2}$ converge to zero as the sample size n tends to infinity. Also note that the unadjusted sample variance $S_{n}^{2}$, despite being biased, has a smaller variance than the adjusted sample variance $s_{n}^{2}$, which is instead unbiased.

Distribution of the estimator

The unadjusted sample variance $S_{n}^{2}$ has a Gamma distribution with parameters $n-1$ and [eq39].

Proof

To prove this result, we need to use some facts on quadratic forms involving normal random variables, which have been introduced in the lecture entitled Normal distribution - Quadratic forms. To understand this proof, you need to first read that lecture, in particular the section entitled Sample variance as a quadratic form. Define the matrix:[eq40]where I is an $n	imes n$ identity matrix and $imath $ is a $n	imes 1$ vector of ones. $M$ is symmetric and idempotent. Denote by X the $n	imes 1 $ random vector whose i-th entry is equal to X_i. The random vector X has a multivariate normal distribution with mean $mu imath $ and covariance matrix $sigma ^{2}I$.

Using the fact that the matrix $M$ is symmetric and idempotent, the unadjusted sample variance can be written as:[eq41]

Using the fact that the random vector[eq42]has a standard multivariate normal distribution and the fact that $Mimath =0$, we can rewrite: [eq43]In other words, $S_{n}^{2}$ is proportional to a quadratic form in a standard normal random vector ($Z^{	op }MZ$) and the quadratic form involves a symmetric and idempotent matrix whose trace is equal to $n-1$. Therefore, the quadratic form $Z^{	op }MZ$ has a Chi-square distribution with $n-1$ degrees of freedom. Finally, we can write:[eq44]i.e. $S_{n}^{2}$ is a Chi-square random variable divided by its number of degrees of freedom and multiplied by [eq45]. Thus, $S_{n}^{2}$ is a Gamma random variable with parameters $n-1$ and [eq46] (see the lecture entitled Gamma distribution for an explanation). Also, by the properties of Gamma random variables, its expected value is:[eq47]and its variance is:[eq48]

The adjusted sample variance $s_{n}^{2}$ has a Gamma distribution with parameters $n-1$ and sigma^2.

Proof

The proof of this result is similar to the proof for unadjusted sample variance found above. It can also be found in the lecture entitled Normal distribution - Quadratic forms. Here, we just notice that $s_{n}^{2}$, being a Gamma random variable with parameters $n-1$ and sigma^2, has expected value[eq49]and variance[eq38]

Risk of the estimator

The mean squared error of the unadjusted sample variance is:[eq51]

Proof

It can be proved as follows:[eq52]

The mean squared error of the adjusted sample variance is:[eq53]

Proof

It can be proved as follows:[eq54]

Therefore the mean squared error of the unadjusted sample variance is always smaller than the mean squared error of the adjusted sample variance:[eq55]

Consistency of the estimator

The unadjusted sample variance[eq28]can be written as:[eq57]where we have defined:[eq58]The two sequences Xbar_n and $W_{n}$ are the sample means of X_n and $X_{n}^{2}$ respectively. The latter both satisfy the conditions of Kolmogorov's Strong Law of Large Numbers (they form IID sequences with finite means), which implies that their sample means Xbar_n and $W_{n}$ converge almost surely to their true means:[eq59]Since the function[eq60]is continuous and almost sure convergence is preserved by continuous transformations, we obtain:[eq61]Therefore the estimator $S_{n}^{2}$ is strongly consistent. It is also weakly consistent, because almost sure convergence implies convergence in probability:[eq62]The adjusted sample variance $s_{n}^{2}$ can be written as:[eq63]The ratio $frac{n}{n-1}$ can be thought of as a constant random variable $Z_{n}$ defined as follows:[eq64]which converges almost surely to 1. Therefore [eq65]where both $Z_{n}$ and $S_{n}^{2}$ are almost surely convergent. Since the product is a continuous function and almost sure convergence is preserved by continuous transformation, we have:[eq66]Thus, also $s_{n}^{2}$ is strongly consistent.

Solved exercises

Below you can find some exercises with explained solutions:

  1. Exercise set 1

The book

Most learning materials found on this website are now available in a traditional textbook format.