1 Welcome to Lab 6!

Intended Learning Outcomes:

  • Introduce and understand the matrix notation of linear models.
  • Obtain model parameters using ordinary least squares (OLS) and using matrix notation in R.

1.1 Introduction to Ordinary Least Squares (OLS)

The primary tool used to model associations among variables is regression (linear or otherwise). Regression analysis is used to model the relationship between a single variable \(Y\), called the response or dependent variable, and one or more explanatory variables, also called predictor(s), covariates, or independent variable(s), \(x_1, x_2,...,x_p\).

As a first step, the response variable will be continuous, but the predictor variables can be either continuous, discrete, or categorical. You can predict non-numeric (continuous) variables using regression models, but they will be called generalised regression models. Here, we stick to the classical regression models. Parameter estimation for linear models can be carried out via OLS or maximum likelihood estimation. In this lab, we will look at OLS.

The word “regression” is due to Sir Francis Galton, who in 1885 demonstrated that offspring do not tend toward the size of the parents; rather, offspring size tends toward the mean of the population. That is, there is a “regression toward mediocrity.”

What else is Francis famous for? "Whenever you can, count", "Nature vs nurture" and "London is the epicenter of female beauty, Aberdeen is the opposite...". Also, eugenics. Fun guy.

1.2 The simplest regression model: simple linear regression (one covariate)

The simple linear regression model is defined as

\[\begin{equation} Y_i = \beta_0 + \beta_1x_i + \epsilon_i, \tag{1.1} \end{equation}\]

where

  • \(Y_i\) is the value of the response variable for the \(i\)th observation.
  • \(\beta_0\) and \(\beta_1\) are regression parameters.
    • \(\beta_0\) is known as the intercept.
    • \(\beta_1\) is the regression coefficient of \(x\) (the effect of \(x\) on \(Y\)).
  • \(\epsilon_i\) is a random error term that is assumed to have mean \(0\) and variance \(sigma^2\).

The "modelling" bit refers to estimating model parameters, \(\beta_0\) and \(\beta_1\) for the \(n\) measurements. It looks like

\[ Y_1 = \beta_0 + \beta_1x_1 + \epsilon_1\\ Y_2 = \beta_0 + \beta_1x_2 + \epsilon_2\\ \vdots\\ Y_n = \beta_0 + \beta_1x_n + \epsilon_n.\\ \] The model can be written in matrix notation as \[ \mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}, \] where

\[\mathbf{Y} = \left[\begin{matrix} Y_1\\Y_2\\ \vdots \\Y_n \end{matrix}\right] \text{ as a } (n \times 1) \text{ matrix, } \mathbf{X} = \left[\begin{matrix} 1 & x_1\\1 & x_2 \\ \vdots&\vdots \\ 1 & x_n \end{matrix}\right] \text{ as } (n \times 2) \text{, } \boldsymbol{\beta} =\left[\begin{matrix} \beta_0 \\ \beta_1 \end{matrix} \right] \text{ as } (2 \times 1) \text{.} \] Note that \(\text{Var}(\boldsymbol{\epsilon}) = \sigma^2\mathbf{I}\) where \(\sigma^2 \mathbf{I}\) is the variance-covariance matrix of the vector of errors. Given that \(\mathbf{I}\) is the identity matrix, \(\sigma^2\) appears only along the diagonals, indicating that \(\epsilon_i\) are independent from each other.

1.3 The next-simplest regression model: multiple linear regression (multiple predictors)

Multiple linear regression is an extension of simple linear regression, where we have more than one coefficient \(x\). \(Y\) is still the dependent variable, \(\beta_0\) the intercept, but \(\beta_1\) is generalised to \(\beta_j\) which is now the coefficient of \(x_{ij}\). The model is written as

\[ Y_i = \beta_0 + \beta_1x_{i1} + \beta_2x_{i2} + \cdots + \beta_{p-1}x_{i(p-1)} + \epsilon_i, \text{ for } i = 1, 2,...,n. \] The model can be written in matrix notation as \[ \mathbf{Y} = \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\epsilon}, \]

where

\[\mathbf{Y} = \left[\begin{matrix} Y_1\\Y2\\ \vdots \\Y_n \end{matrix}\right] \text{ as a } (n \times 1) \text{ matrix, } \mathbf{X} = \left[\begin{matrix} 1 & x_{11} & \cdots& x_{1(p-1)}\\1 & x_{21} & \cdots& x_{2(p-1)} \\ \vdots&\vdots&&\vdots \\ 1 & x_{n1} & \cdots& x_{n(p-1)} \end{matrix}\right] \text{ as } (n \times p) \text{, } \boldsymbol{\beta} =\left[\begin{matrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_n \end{matrix} \right] \text{ as } (p \times 1), \text{ and } \mathbf{\epsilon} = \left[\begin{matrix} \epsilon_0 \\ \epsilon_1 \\ \vdots \\ \epsilon_n \end{matrix} \right] \text{ as } (n \times 1) \text{.} \]

Note some things

  • \(\mathbf{X}\) is a matrix of known constants, containing the values of particular predictors.
  • \(\mathbf{Y}\) and \(\mathbf{\epsilon}\) are random vectors whose elements are random variables.
  • \(\boldsymbol{\beta}\) is a vector of unknown coefficients (what we have to estimate!).

1.4 Ordinary least squares

The ordinary least squares method of estimating parameters minimises the sum of the squared deviations of the \(Y_i\)s from their expected values such that

\[\begin{split} \epsilon_i &= Y_i - E(Y_i)\\ &= Y_i - (\beta_0 + \beta_1 x_i)\quad \text{in the case of simple linear regression}. \end{split}\]

The estimates \(\hat{\beta}_0\) of \(\beta_0\) and \(\hat{\beta_1}\) of \(\beta_1\) seek to minimise the quantity \(\mathcal{Q}\), which represents the sum of the squared residuals (differences) as

\[ \mathcal{Q} = \sum_{i=1}^{n}\epsilon_i^2 = \sum_{i=1}^n\left((Y_i - (\beta_0 + \beta_{1}x_i)\right)^2, \] which, in matrix notation, is the same as saying \[ \mathcal{Q} = \epsilon'\epsilon = (\mathbf{Y} - \mathbf{X}\boldsymbol{\beta})'(\mathbf{Y} - \mathbf{X}\boldsymbol{\beta}). \] If we want parameter estimates that minimise \(\mathcal{Q}\), we take the derivative with respect to our parameters

\[ \begin{aligned} \frac{\delta \mathcal{Q}}{\delta \beta_0} & =2 \sum_{i=1}^n\left(Y_i-\beta_0-\beta_1 x_i\right)(-1) \\ & =-2 \sum_{i=1}^n\left(Y_i-\beta_0-\beta_1 x_i\right) . \\ \frac{\delta \mathcal{Q}}{\delta \beta_1} & =2 \sum_{i=1}^n\left(Y_i-\beta_0-\beta_1 x_i\right)\left(-x_i\right) \\ & =-2 \sum_{i=1}^n\left(Y_i-\beta_0-\beta_1 x_i\right)\left(x_i\right), \end{aligned} \] set it to zero, and solve for \(\hat{\beta}_j\). The solutions for \(\hat{\beta}_0\) and \(\hat{\beta}_1\) are

\[ \begin{aligned} \hat{\beta}_0 & = \overline{Y} - \hat{\beta}_1\overline{x} \\ \hat{\beta}_1 & = \frac{\sum_{i=1}^n(x_i - \overline{x})(Y_i - \overline{Y})}{\sum_{i=1}^{n}(x_i - \overline{x})^2}. \end{aligned} \]

In matrix notation, we can generalise the estimation of \(\boldsymbol{\beta}\) as

\[\begin{equation} \hat{\boldsymbol{\beta}} = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{Y}. \tag{1.2} \end{equation}\]