Search for probability and statistics terms on Statlect
StatLect
Index > Fundamentals of statistics

The normal linear regression model

by , PhD

This lecture discusses the main properties of the Normal Linear Regression Model (NLRM), a linear regression model in which the vector of errors of the regression is assumed to have a multivariate normal distribution conditional on the matrix of regressors. The assumption of multivariate normality, together with other assumptions (mainly concerning the covariance matrix of the errors), allows us to derive analytically the distributions of the Ordinary Least Squares (OLS) estimators of the regression coefficients and of several other statistics.

Table of Contents

Setting

We use the same notation used in the lecture entitled Properties of the OLS estimator (to which you can refer for more details): the $N imes 1$ vector of observations of the dependent variable is denoted by $y$, the $N imes K$ matrix of regressors (called design matrix) is denoted by X, the $N imes 1 $ vector of errors is denoted by epsilon and the Kx1 vector of regression coefficients is denoted by $beta $, so that the regression equations can be written in matrix form as[eq1]The OLS estimator $widehat{beta }$ is the vector which minimizes the sum of squared residuals[eq2]and, if the design matrix X has full rank, it can be computed as[eq3]

Assumptions

The assumptions made in a normal linear regression model are:

  1. the design matrix X has full-rank (as a consequence, $X^{ op }X$ is invertible and the OLS estimator is [eq4]);

  2. conditional on X, the vector of errors epsilon has a multivariate normal distribution with mean equal to 0 and covariance matrix equal to[eq5]where sigma^2 is a positive constant and I is the $N imes N$ identity matrix;

Note that the assumption that the covariance matrix of epsilon is diagonal implies that the entries of epsilon are mutually independent, that is, $varepsilon _{i}$ is independent of $varepsilon _{j}$ for $i eq j$. Moreover, the assumption that all diagonal entries of the covariance matrix are equal implies that all the entries of epsilon have the same variance, that is, [eq6] for any i. The latter assumption is often referred to as "homoscedasticity assumption", and if the assumption is satisfied, we say that the errors are homoscedastic. On the contrary, if homoscedasticity does not hold, we say that the errors are heteroscedastic.

Distribution of the OLS estimator

Under the assumptions made in the previous section, the OLS estimator has a multivariate normal distribution, conditional on the design matrix.

Proposition In a Normal Linear Regression Model, the OLS estimator $widehat{beta }$ has a multivariate normal distribution, conditional on X, with mean [eq7]and covariance matrix[eq8]

Proof

First of all, note that[eq9]The fact that we are conditioning on X means that we can treat X as a constant matrix. Therefore, conditional on X, the OLS estimator $widehat{beta }$ is a linear transformation of a multivariate normal random vector (the vector epsilon). This implies that also $widehat{beta }$ is multivariate normal, with mean[eq10]and variance[eq11]

Note that [eq12] means that the OLS estimator is unbiased, not only conditionally, but also unconditionally, because by the Law of Iterated Expectations we have that[eq13]

Estimation of the variance of the error terms

The variance of the error terms sigma^2 is usually not known. A commonly used estimator of sigma^2 is the adjusted sample variance of the residuals:[eq14]where the regression residuals are [eq15]

The properties enjoyed by [eq16] are summarized by the following proposition.

Proposition In a Normal Linear Regression Model, the adjusted sample variance of the residuals [eq16] is a conditionally unbiased estimator of sigma^2:[eq18]Furthermore, conditional on X, [eq16] has a Gamma distribution with parameters $N-K$ and sigma^2 and it is independent of $widehat{beta }$.

Proof

Denote by $e$ the $N imes 1$ vector of residuals. Remember from the previous proof that the OLS estimator can be written as[eq20]As a consequence, we have[eq21]The matrix[eq22]is clearly symmetric (verify it by taking its transpose). It is also idempotent because[eq23]Therefore,[eq24]where [eq25] has a standard multivariate normal distribution, that is, a multivariate normal distribution with zero mean and unit covariance matrix. Since the matrix $M$ is symmetric and idempotent, the quadratic form [eq26]has a Chi-square distribution with a number of degrees of freedom equal to the trace of the matrix $M$ (see the lecture Normal distribution - Quadratic forms). But the trace of $M$ is[eq27]Since the expected value of a Chi-square random variable is equal to its number of degrees of freedom, we have[eq28]Moreover, the fact that the quadratic form $Q$ has a Chi-square distribution with $N-K$ degrees of freedom implies that the sample variance[eq29]has a Gamma distribution with parameters $N-K$ and sigma^2 (see the lecture on the Gamma distribution for a proof of this fact). To conclude, we need to prove that [eq30] is independent of $widehat{beta }$. Since[eq31]and[eq32]we have that [eq16] and $widehat{beta }$ are functions of the same multivariate normal random vector epsilon. Therefore, by standard results on the independence of quadratic forms involving normal vectors, [eq16] and $widehat{beta }$ are independent if $Mvarepsilon $ and [eq35] are orthogonal. In order to check their orthogonality, we only need to verify that the product between [eq36] and $M$ is zero:[eq37]

Note that also in this case, the proposed estimator is unbiased not only conditionally, but also unconditionally because, by the Law of Iterated Expectations, we have that[eq38]

Estimation of the covariance matrix of the OLS estimator

We have already proved that in the Normal Linear Regression Model the conditional covariance matrix of the OLS estimator (conditional on X) is[eq8]

In practice, however, this quantity is not known exactly because the variance of the error terms, that is sigma^2, is unknown. However, we can replace its unknown value with the estimator proposed above (the adjusted sample variance of the residuals), so as to obtain an estimator of the covariance matrix of $widehat{beta }$:[eq40]

This estimator is often employed to construct test statistics that allow us to conduct tests of hypotheses about the regression coefficients.

Maximum likelihood estimation

It can be proved that the OLS estimators of the coefficients of a Normal Linear Regression Model are equal to the maximum likelihood estimators. On the contrary, the maximum likelihood estimator of the variance of the error terms is different from the estimator derived above. For proofs of these two facts, see the lecture entitled Linear Regression - Maximum likelihood estimation.

Hypothesis testing

In the lecture on Linear regressions and hypothesis testing we explain how to perform hypothesis tests on the coefficients of a normal linear regression model.

How to cite

Please cite as:

Taboga, Marco (2021). "The normal linear regression model", Lectures on probability theory and mathematical statistics. Kindle Direct Publishing. Online appendix. https://www.statlect.com/fundamentals-of-statistics/normal-linear-regression-model.

The books

Most of the learning materials found on this website are now available in a traditional textbook format.

Featured pages
Explore
Main sections
About
Glossary entries
Share
  • To enhance your privacy,
  • we removed the social buttons,
  • but don't forget to share.