In linear regression, our fundamental objective is to explain the variability observed in the outcome variable (\(Y\)) using one or more explanatory variable (\(X_1, X_2, \dots, X_k\)).
\[ \hat{Y}_i = b_0 + b_1 X_{1i} + b_2 X_{2i} + \dots + b_k X_{ki} \]
It is crucial to know
\[ (Y_i - \hat{Y}_i) = (Y_i - \bar{Y})-(\hat{Y}_i - \bar{Y}) \]
Where:
Squaring both sides and summing across all \(n\) observations yields the fundamental sum-of-squares identity:
\[ \text{SSE} = \text{SST} - \text{SSR} \]
Where
| Metric | Formula | Degrees of Freedom (\(df\)) | Conceptual Intuition |
|---|---|---|---|
| SST | \(\sum_{i=1}^n (Y_i - \bar{Y})^2\) | \(n - 1\) | Total variation in the outcome \(Y\) without using any predictors. |
| SSR | \(\sum_{i=1}^n (\hat{Y}_i - \bar{Y})^2\) | \(k\) | Variation accounted by the regression model. |
| SSE | \(\sum_{i=1}^n (Y_i - \hat{Y}_i)^2\) \(= \sum_{i=1}^n e_i^2\) |
\(n - k - 1\) | Unexplained variation; sum of squared discrepancies between observed points and regression surface. |
\(R^2\) (The coefficient of determination) quantifies the proportion of total variation in the response variable explained by the linear combination of predictors:
\[ R^2 = \frac{\text{SSR}}{\text{SST}} = 1 - \frac{\text{SSE}}{\text{SST}} \]
Interpretation Rules
The Adjusted \(R^2\) incorporates degrees of freedom to penalize excessive parameterization and prevent overfitting, :
\[ R^2_{\text{adj}} = 1 - \left[ \frac{\text{SSE} / (n - k - 1)}{\text{SST} / (n - 1)} \right] = 1 - \left( \frac{\text{MSE}}{\text{MST}} \right) \]
Alternatively expressed via \(R^2\): \[ R^2_{\text{adj}} = 1 - (1 - R^2) \frac{n - 1}{n - k - 1} \]
Key Properties
\(F\)-Test** evaluates whether the collection of predictors collectively explains a statistically significant amount of variation.
Hypotheses
\[ H_0: \beta_1 = \beta_2 = \dots = \beta_k = 0 \]
\[ H_1: \text{There is at least one} \beta_j \neq 0 \]
Difference on individual testing on \(\beta_i\)
Before calculating the \(F\)-statistic, we normalize the sums of squares by their respective degrees of freedom:
Mean Square Regression (MSR):
\[ \text{MSR} = \frac{\text{SSR}}{k} \]
Mean Square Error (MSE):
\[ \text{MSE} = \frac{\text{SSE}}{n - k - 1} = s_e^2 \quad (\text{an unbiased estimator of } \sigma^2) \]
The \(F\)-statistic is the ratio of explained variance per degree of freedom to unexplained variance per degree of freedom:
\[ F = \frac{\text{MSR}}{\text{MSE}} = \frac{\text{SSR} / k}{\text{SSE} / (n - k - 1)} \]
It can also be written directly in terms of \(R^2\):
\[ F = \frac{R^2 / k}{(1 - R^2) / (n - k - 1)} \]
Under the null hypothesis \(H_0\), and assuming normally distributed homoscedastic errors: \[ F \sim F(df_1 = k, \, df_2 = n - k - 1) \]
Statistical software (e.g., R, Python statsmodels, Stata, SAS) formats these calculations into an ANOVA table:
| Source of Variation | Sum of Squares (\(SS\)) | Degrees of Freedom (\(df\)) | Mean Square (\(MS\)) | \(F\)-Statistic | \(p\)-value |
|---|---|---|---|---|---|
| Regression (Model) | \(\text{SSR}\) | \(k\) | \(\text{MSR} = \frac{\text{SSR}}{k}\) | \(F = \frac{\text{MSR}}{\text{MSE}}\) | \(P(F_{(k, n-k-1)} \ge F)\) |
| Residual (Error) | \(\text{SSE}\) | \(n - k - 1\) | \(\text{MSE} = \frac{\text{SSE}}{n - k - 1}\) | — | — |
| Total | \(\text{SST}\) | \(n - 1\) | \(\text{MST} = \frac{\text{SST}}{n - 1}\) | — | — |
Research purpose: Factors affecting infant birth weight (\(n = 189\)).
# Load MASS (pre-installed in R)
library(MASS)
data(birthwt)
# Convert binary indicator variables into explicit factors
birthwt$smoke <- factor(birthwt$smoke, labels = c("Non-Smoker", "Smoker"))
birthwt$ht <- factor(birthwt$ht, labels = c("No", "Yes"))
birthwt$ui <- factor(birthwt$ui, labels = c("No", "Yes"))Residuals:
Min 1Q Median 3Q Max
-1675.13 -464.38 -1.02 459.70 1779.69
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 2506.354 291.759 8.591 3.73e-15 ***
age 3.648 9.432 0.387 0.69935
lwt 4.388 1.691 2.594 0.01024 *
smokeSmoker -240.848 100.413 -2.399 0.01746 *
htYes -643.958 207.232 -3.107 0.00219 **
uiYes -547.068 139.937 -3.909 0.00013 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 671.4 on 183 degrees of freedom
Multiple R-squared: 0.1747, Adjusted R-squared: 0.1522
F-statistic: 7.75 on 5 and 183 DF, p-value: 1.234e-06 ANOVA
------------------------------------------------------------------------
Sum of
Squares DF Mean Square F Sig.
------------------------------------------------------------------------
Regression 17469477.719 5 3493895.544 7.75 0.0000
Residual 82500178.091 183 450820.645
Total 99969655.810 188
------------------------------------------------------------------------\(R^2\) Calculation: \[ R^2 = \frac{17469477.719 }{ 99969655.810} = 17.4\% \] Interpretation: 12.6% of the variance in biomarker reduction is explained by maternal weight at last menstrual period, baseline metabolic index, and age.
Adjusted \(R^2\) Calculation: \[ R^2_{\text{adj}} = 1 - \left( \frac{450820.645}{99969655.810/ 188} \right) = = 1 - 0.8478 = 15.2\% \] Interpretation: After penalizing for adding three model parameters, the model accounts for approximately 70.2% of the variance.
\(F\)-Test Evaluation: \(F_{(5, 183)} = 7.75, \quad p < 0.0001\) Conclusion: Since the \(p\)-value is well below \(\alpha = 0.05\), we reject \(H_0\). The model as a whole explains a statistically significant amount of variation compared to a flat horizontal line.
\(\beta\)
In simple linear regression (\(k = 1\)): \[ F_{(1, n-2)} = (t_{\beta_1})^2 \]
The two-sided \(t\)-test on the single slope coefficient \(\beta_1\) and the overall \(F\)-test are mathematically identical and yield the exact same \(p\)-value.
This classic paradox typically occurs due to severe multicollinearity.
[Fit OLS Regression Model]
│
▼
[Check Overall F-Test]
├── p >= 0.05 ───► Model does not explain variance beyond noise. Stop / Respecify.
└── p < 0.05 ────► Model explains significant variance. Proceed:
│
┌───────────────┴──────────────┐
▼ ▼
[Assess Fit] [Inspect Coefficients]
• Check R² & Adj R² • Examine t-tests & p-values
• Check Root MSE (s_e) • Check signs & magnitude
• Check VIF for multicollinearity
│
▼
[Validate Assumptions]
• Residual vs Fitted (Linearity / Homoscedasticity)
• Normal Q-Q Plot (Normality of residuals)
• Leverage / Cook's D (Influential outliers)