Model Comparisons

🛠 Nested F-test for Multiple Linear Regression

• What does “nested” mean?
  • You have a “small” model and a “large” model where the “small” model is completely contained in the “large” model
• The F-test we have learned so far is one example of this, comparing:
  • \(y = \beta_0 + \epsilon\) (small)
  • \(y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \dots +\beta_kx_k + \epsilon\) (large)
• The full (large) model has \(k\) predictors, the reduced (small) model has \(k - p\) predictors

🛠 Nested F-test for Multiple Linear Regression

• The full (large) model has \(k\) predictors, the reduced (small) model has \(k - p\) predictors
• What is \(H_0\)?
  • \(H_0:\) \(\beta_i=0\) for all \(p\) predictors being dropped from the full model
• What is \(H_A\)?
  • \(H_A:\) \(\beta_i\neq 0\) for at least one of the \(p\) predictors dropped from the full model
• Does the full model do a (statistically significant) better job of explaining the variability in the response than the reduced model?

🛠 Nested F-test for Multiple Linear Regression

data("Perch")
model1 <- lm(
  Weight ~ Length + Width + Length * Width,
  data = Perch
  )
model2 <- lm(
  Weight ~ Length + Width + Length * Width + I(Length ^ 2) + I(Width ^ 2),
  data = Perch
  )

• If we want to do a nested F-test, what is \(H_0\)?
  • \(H_0: \beta_4 = \beta_5 = 0\)
• What is \(H_A\)?
  • \(H_A: \beta_4\neq 0\) or \(\beta_5\neq 0\)
• What are the degrees of freedom of this test? (n = 56)
  • 2, 50

🛠 \(R^2\) for Multiple Linear Regression

• \(R^2= \frac{SSModel}{SSTotal}\)
• \(R^2= 1 - \frac{SSE}{SSTotal}\)
• As is, if you add a predictor this will always increase. Therefore, we have \(R^2_{adj}\) that has a small “penalty” for adding more predictors
• \(R^2_{adj} = 1 - \frac{SSE/(n-k-1)}{SSTotal / (n-1)}\)
• \(\frac{SSTotal}{n-1} = \frac{\sum(y - \bar{y})^2}{n-1}\) What is this?
  • Sample variance! \(S_Y^2\)
• \(R^2_{adj} = 1 - \frac{\hat\sigma^2_\epsilon}{S_Y^2}\)

🛠 \(R^2_{adj}\) for Multiple Linear Regression

• \(R^2_{adj} = 1 - \frac{SSE/(n-k-1)}{SSTotal / (n-1)}\)
• The denominator stays the same for all models fit to the same response variable and data
• the numerator actually increase when a new predictor is added to a model if the decrease in the SSE is not sufficient to offset the decrease in the error degrees of freedom.
• So \(R^2_{adj}\) can 👇 when a weak predictor is added to a model

Log Likelihood

• Both AIC and BIC are calculated using the log likelihood
• \(\log(\mathcal{L}) = -\frac{n}{2}[\log(2\pi) + \log(SSE/n) + 1]\)
  • \(\log = \log_e\), log() in R

logLik(model1)

'log Lik.' -290 (df=5)

-56 / 2 * (log(2 * pi) + log(101765 / 56) + 1)

[1] -290

Log Likelihood

What I want you to remember

• Both AIC and BIC are calculated using the log likelihood

\[\log(\mathcal{L}) = -\frac{n}{2}[\log(SSE/n) ]+\textrm{some constant}\]

• \(\log = \log_e\), log() in R
• “goodness of fit” measure
• higher log likelihood is better

AIC

• Akaike’s Information Criterion
• \(AIC = 2(k+1) - 2\log(\mathcal{L})\)
• \(k\) is the number of predictors in the model
• lower AIC values are better

AIC(model1)

[1] 589

AIC(model2)

[1] 588

BIC

• Bayesian Information Criterion
• \(BIC = \log(n)(k+1) - 2\log(\mathcal{L})\)
• \(k\) is the number of predictors in the model
• lower BIC values are better

BIC(model1)

[1] 599

BIC(model2)

[1] 602

AIC and BIC can disagree!

AIC(model1)

[1] 589

AIC(model2)

[1] 588

BIC(model1)

[1] 599

BIC(model2)

[1] 602

• the penalty term is larger in BIC than in AIC
• What to do? Both are valid, pre-specify which you are going to use before running your models in the methods section of your analysis