The Least-squares Solution - Revision history

Khanh at 17:07, 4 November 2021

2021-11-04T17:07:03Z

Lila at 20:51, 12 October 2021

2021-10-12T20:51:44Z

Lila: /* Matrix/vector formulation */

2021-10-12T20:50:02Z

Matrix/vector formulation

Lila: /* Linear model */

2021-10-12T20:49:28Z

Linear model

Lila: Created page with "'''Ordinary least squares''' ('''OLS''') is a type of linear least squares method for estimating the unknown parameters in a linear regression model. OLS chooses the parameter..."

2021-10-12T20:49:01Z

Created page with "'''Ordinary least squares''' ('''OLS''') is a type of linear least squares method for estimating the unknown parameters in a linear regression model. OLS chooses the parameter..."

New page

'''Ordinary least squares''' ('''OLS''') is a type of linear least squares method for estimating the unknown parameters in a linear regression model. OLS chooses the parameters of a linear function of a set of explanatory variables by the principle of least squares: minimizing the sum of the squares of the differences between the observed dependent variable (values of the variable being observed) in the given dataset and those predicted by the linear function of the independent variable.

Geometrically, this is seen as the sum of the squared distances, parallel to the axis of the dependent variable, between each data point in the set and the corresponding point on the regression surface—the smaller the differences, the better the model fits the data. The resulting estimator can be expressed by a simple formula, especially in the case of a simple linear regression, in which there is a single regressor on the right side of the regression equation.

The OLS estimator is consistent when the regressors are exogenous, and—by the Gauss–Markov theorem-optimal in the class of linear unbiased estimators when the errors are homoscedastic and serially uncorrelated. Under these conditions, the method of OLS provides minimum-variance mean-unbiased estimation when the errors have finite variances. Under the additional assumption that the errors are normally distributed, OLS is the maximum likelihood estimator.

== Linear model ==

[[File:Okuns law quarterly differences.svg|300px|thumb|Okun's law in macroeconomics states that in an economy the GDP growth should depend linearly on the changes in the unemployment rate. Here the ordinary least squares method is used to construct the regression line describing this law.]]
Suppose the data consists of <math>n</math> observations <math>\left\{\mathbf{x}_i, y_i\right\}_{i=1}^n</math>. Each observation <math>i</math> includes a scalar response <math>y_i</math> and a column vector <math>\mathbf{x}_i</math> of <math>p</math> parameters (regressors), i.e., <math>\mathbf{x}_i=\left[x_{i1}, x_{i2}, \dots, x_{ip}\right]^\mathsf{T}</math>. In a linear regression model, the response variable, <math>y_i</math>, is a linear function of the regressors:

:<math>y_i = \beta_1\ x_{i1} + \beta_2\ x_{i2} + \cdots + \beta_p\ x_{ip} + \varepsilon_i,</math>

or in vector form,
: <math> y_i = \mathbf{x}_i^\mathsf{T} \boldsymbol{\beta} + \varepsilon_i, \, </math>
where <math>\mathbf{x}_i</math>, as introduced previously, is a column vector of the <math>i</math>-th observation of all the explanatory variables; <math>\boldsymbol{\beta}</math> is a <math>p \times 1</math> vector of unknown parameters; and the scalar <math>\varepsilon_i</math> represents unobserved random variables ([[errors and residuals in statistics|errors]]) of the <math>i</math>-th observation. <math>\varepsilon_i</math> accounts for the influences upon the responses <math>y_i</math> from sources other than the explanators <math>\mathbf{x}_i</math>. This model can also be written in matrix notation as
: <math> \mathbf{y} = \mathrm{X} \boldsymbol{\beta} + \boldsymbol{\varepsilon}, \, </math>

where <math>\mathbf{y}</math> and <math>\boldsymbol{\varepsilon}</math> are <math>n \times 1</math> vectors of the response variables and the errors of the <math> n </math> observations, and <math>\mathrm{X}</math> is an <math>n \times p</math> matrix of regressors, also sometimes called the [[design matrix]], whose row <math>i</math> is <math>\mathbf{x}_i^\mathsf{T}</math> and contains the <math>i</math>-th observations on all the explanatory variables.

As a rule, the constant term is always included in the set of regressors <math>\mathrm{X}</math>, say, by taking <math>x_{i1}=1</math> for all <math>i=1, \dots, n</math>. The coefficient <math>\beta_1</math> corresponding to this regressor is called the ''intercept''.

Regressors do not have to be independent: there can be any desired relationship between the regressors (so long as it is not a linear relationship). For instance, we might suspect the response depends linearly both on a value and its square; in which case we would include one regressor whose value is just the square of another regressor. In that case, the model would be ''quadratic'' in the second regressor, but none-the-less is still considered a ''linear'' model because the model ''is'' still linear in the parameters (<math>\boldsymbol{\beta}</math>).

===Matrix/vector formulation===
Consider an overdetermined system

:<math>\sum_{j=1}^{p} X_{ij} \beta_j = y_i,\ (i=1, 2, \dots, n),</math>

of <math> n </math> linear equations in <math>p</math> unknown coefficients, <math> \beta_1, \beta_2, \dots, \beta_p </math>, with <math> n > p </math>. (Note: for a linear model as above, not all elements in <math> \mathrm{X} </math> contains information on the data points. The first column is populated with ones, <math>X_{i1} = 1</math>. Only the other columns contain actual data. So here <math>p</math> is equal to the number of regressors plus one.) This can be written in matrix form as

:<math>\mathrm{X} \boldsymbol{\beta} = \mathbf {y},</math>

where

:<math>\mathrm{X}=\begin{bmatrix}
X_{11} & X_{12} & \cdots & X_{1p} \\
X_{21} & X_{22} & \cdots & X_{2p} \\
\vdots & \vdots & \ddots & \vdots \\
X_{n1} & X_{n2} & \cdots & X_{np}
\end{bmatrix} ,
\qquad \boldsymbol \beta = \begin{bmatrix}
\beta_1 \\ \beta_2 \\ \vdots \\ \beta_p \end{bmatrix} ,
\qquad \mathbf y = \begin{bmatrix}
y_1 \\ y_2 \\ \vdots \\ y_n
\end{bmatrix}. </math>

Such a system usually has no exact solution, so the goal is instead to find the coefficients <math>\boldsymbol{\beta}</math> which fit the equations "best", in the sense of solving the quadratic minimization problem

:<math>\hat{\boldsymbol{\beta}} = \underset{\boldsymbol{\beta}}{\operatorname{arg\,min}}\,S(\boldsymbol{\beta}), </math>

where the objective function <math> S </math> is given by

:<math>S(\boldsymbol{\beta}) = \sum_{i=1}^n \biggl| y_i - \sum_{j=1}^p X_{ij}\beta_j\biggr|^2 = \bigl\|\mathbf y - \mathrm{X} \boldsymbol \beta \bigr\|^2.</math>

A justification for choosing this criterion is given in Properties below. This minimization problem has a unique solution, provided that the <math>p</math> columns of the matrix <math> \mathrm{X} </math> are linearly independent, given by solving the normal equation

:<math>(\mathrm{X}^{\mathsf T} \mathrm{X} )\hat{\boldsymbol{\beta}} = \mathrm{X}^{\mathsf T} \mathbf y\ .</math>

The matrix <math>\mathrm{X}^{\mathsf T} \mathrm X</math> is known as the [[Gram matrix]] and the matrix <math>\mathrm{X}^{\mathsf T} \mathbf y</math> is known as the moment matrix of regressand by regressors. Finally, <math>\hat{\boldsymbol{\beta}}</math> is the coefficient vector of the least-squares hyperplane, expressed as

:<math>\hat{\boldsymbol{\beta}}= \left( \mathrm{X}^{\mathsf T} \mathrm{X} \right)^{-1} \mathrm{X}^{\mathsf T} \mathbf y.</math>

or

:<math>\hat{\boldsymbol{\beta}}= \boldsymbol{\beta} + (\mathbf{X}^\top \mathbf{X} )^{-1}\mathbf {X}^\top \boldsymbol{\varepsilon}.</math>

@@ Line 66: / Line 66: @@
 :<math>\hat{\boldsymbol{\beta}}= \boldsymbol{\beta} + (\mathbf{X}^\top \mathbf{X} )^{-1}\mathbf {X}^\top \boldsymbol{\varepsilon}.</math>
-==Resources==
+== Licensing ==
+Content obtained and/or adapted from:
-* [https://en.wikipedia.org/wiki/Ordinary_least_squares Ordinary least squares], Wikipedia
+* [https://en.wikipedia.org/wiki/Ordinary_least_squares Ordinary least squares, Wikipedia] under a CC BY-SA license

← Older revision		Revision as of 20:51, 12 October 2021
Line 65:		Line 65:

	:<math>\hat{\boldsymbol{\beta}}= \boldsymbol{\beta} + (\mathbf{X}^\top \mathbf{X} )^{-1}\mathbf {X}^\top \boldsymbol{\varepsilon}.</math>		:<math>\hat{\boldsymbol{\beta}}= \boldsymbol{\beta} + (\mathbf{X}^\top \mathbf{X} )^{-1}\mathbf {X}^\top \boldsymbol{\varepsilon}.</math>
		+
		+	==Resources==
		+
		+	* [https://en.wikipedia.org/wiki/Ordinary_least_squares Ordinary least squares], Wikipedia

@@ Line 58: / Line 58: @@
 :<math>(\mathrm{X}^{\mathsf T} \mathrm{X} )\hat{\boldsymbol{\beta}} = \mathrm{X}^{\mathsf T} \mathbf y\ .</math>
-The matrix <math>\mathrm{X}^{\mathsf T} \mathrm X</math> is known as the [[Gram matrix]] and the matrix <math>\mathrm{X}^{\mathsf T} \mathbf y</math> is known as the moment matrix of regressand by regressors. Finally, <math>\hat{\boldsymbol{\beta}}</math> is the coefficient vector of the least-squares hyperplane, expressed as
+The matrix <math>\mathrm{X}^{\mathsf T} \mathrm X</math> is known as the Gram matrix and the matrix <math>\mathrm{X}^{\mathsf T} \mathbf y</math> is known as the moment matrix of regressand by regressors. Finally, <math>\hat{\boldsymbol{\beta}}</math> is the coefficient vector of the least-squares hyperplane, expressed as
 :<math>\hat{\boldsymbol{\beta}}= \left( \mathrm{X}^{\mathsf T} \mathrm{X} \right)^{-1} \mathrm{X}^{\mathsf T} \mathbf y.</math>

@@ Line 17: / Line 17: @@
 : <math> \mathbf{y} = \mathrm{X} \boldsymbol{\beta} + \boldsymbol{\varepsilon}, \,  </math>
-where <math>\mathbf{y}</math> and <math>\boldsymbol{\varepsilon}</math> are <math>n \times 1</math> vectors of the response variables and the errors of the <math> n  </math> observations, and <math>\mathrm{X}</math> is an <math>n \times p</math> matrix of regressors, also sometimes called the [[design matrix]], whose row <math>i</math> is <math>\mathbf{x}_i^\mathsf{T}</math> and contains the <math>i</math>-th observations on all the explanatory variables.
+where <math>\mathbf{y}</math> and <math>\boldsymbol{\varepsilon}</math> are <math>n \times 1</math> vectors of the response variables and the errors of the <math> n  </math> observations, and <math>\mathrm{X}</math> is an <math>n \times p</math> matrix of regressors, also sometimes called the design matrix, whose row <math>i</math> is <math>\mathbf{x}_i^\mathsf{T}</math> and contains the <math>i</math>-th observations on all the explanatory variables.
 As a rule, the constant term is always included in the set of regressors <math>\mathrm{X}</math>, say, by taking <math>x_{i1}=1</math> for all <math>i=1, \dots, n</math>. The coefficient <math>\beta_1</math> corresponding to this regressor is called the ''intercept''.