Yu Cheng Hsu – Regression diagnostics

Learning objectives

Demonstrate linear regression diagnostics
Judgement on the performance of linear regression

Overview of linear regression

In linear regression, our fundamental objective is to explain the variability observed in the outcome variable ($Y$) using one or more explanatory variable ($X_1, X_2, \dots, X_k$).

It is crucial to know

Does $X_1, X_2, \dots, X_k$ really make explaination to $Y$
How good is the model performance?

Evaluating linear regression - from an observation

The regression diagnostics lies in the Analysis of Variance (ANOVA)
We can decompose the error of you prediction as:

Where:

$(Y_i - \hat{Y}_i) = e_i$ = Unexplained residual deviation (the error remaining after fitting the model).
$(Y_i - \bar{Y})$ = Total deviation of the data point from the baseline benchmark (sample mean).
$(\hat{Y}_i - \bar{Y})$ = Explained deviation (the difference between the model’s fitted value and the sample mean).

A summary of error

Squaring both sides and summing across all $n$ observations yields the fundamental sum-of-squares identity:

Where

SST (Total Sum of Squares): $\sum_i (Y_i - \hat{Y}_i)^2$
SSE (Error Sum of Squares): $\sum_i (Y_i - \bar{Y})^2$
SSR (Regression Sum of Squares): $\sum_i (\hat{Y}_i - \bar{Y})^2$

Mathematical Formulations & Intuitions

Metric	Formula	Degrees of Freedom ($df$)	Conceptual Intuition
SST	$\sum_{i=1}^n (Y_i - \bar{Y})^2$	$n - 1$	Total baseline variation in the outcome $Y$ without using any predictors.
SSR	$\sum_{i=1}^n (\hat{Y}_i - \bar{Y})^2$	$k$	Variation successfully accounted for by the regression model.
SSE	$\sum_{i=1}^n (Y_i - \hat{Y}_i)^2$ $= \sum_{i=1}^n e_i^2$	$n - k - 1$	Unexplained variation; sum of squared discrepancies between observed points and regression surface.

Goodness-of-Fit: $R^2$

$R^2$ (The coefficient of determination) quantifies the proportion of total variation in the response variable explained by the linear combination of predictors:

Interpretation Rules

Range: In standard OLS with an intercept, $0 \le R^2 \le 1$ (or $0\%$ to $100\%$).
Interpretation: If $R^2 = 0.74$, exactly 74% of the variability in $Y$ is explained by the predictors included in the model, while the remaining 26% is attributable to residual noise or omitted factors.
Limitation: $R^2$ is non-decreasing with respect to the addition of predictors. Adding completely unrelated random noise variables will force $\text{SSE}$ to stay equal or decrease, artificially inflating $R^2$.

Adjusted $R^2$ ($R^2_{\text{adj}}$)

The Adjusted $R^2$ incorporates degrees of freedom to penalize excessive parameterization and prevent overfitting, :

Alternatively expressed via $R^2$: \[ R^2_{\text{adj}} = 1 - (1 - R^2) \frac{n - 1}{n - k - 1} \]

Key Properties

Increases only if a newly added predictor reduces $\text{SSE}$ by a margin greater than what would be expected by pure chance.
Can occasionally yield a negative value when model fit is negligible; in such cases, it is effectively interpreted as zero.

$F$-Test for Model Significance

$F$-Test** evaluates whether the collection of predictors collectively explains a statistically significant amount of variation.

Hypotheses

Null Hypothesis ($H_0$): All slope coefficients are simultaneously zero.

Alternative Hypothesis ($H_1$): At least one slope coefficient is non-zero.

Difference on individual testing on $\beta_i$

$\beta_i$ is a $t$-test
Repeated testing $\beta_i$ cause p-value inflation

Mean Squares: Variance Between vs. Variance Within

Before calculating the $F$-statistic, we normalize the sums of squares by their respective degrees of freedom:

Mean Square Regression (MSR):

Mean Square Error (MSE):

Constructing the $F$-Statistic

The $F$-statistic is the ratio of explained variance per degree of freedom to unexplained variance per degree of freedom:

It can also be written directly in terms of $R^2$:

\[ F = \frac{R^2 / k}{(1 - R^2) / (n - k - 1)} \]

Sampling Distribution & Decision Rule:

Under the null hypothesis $H_0$, and assuming normally distributed homoscedastic errors: \[ F \sim F(df_1 = k, \, df_2 = n - k - 1) \]

Decision: Reject $H_0$ at significance level $\alpha$ if $F > F_{\text{crit}}(\alpha, k, n - k - 1)$ or if $p\text{-value} < \alpha$.
Conclusion: Rejecting $H_0$ confirms that the regression model explains significantly more variance than a naive “intercept-only” model (sample mean $\bar{Y}$).

5. The Standard ANOVA Table for Regression

Statistical software (e.g., R, Python statsmodels, Stata, SAS) formats these calculations into an ANOVA table:

Source of Variation	Sum of Squares ($SS$)	Degrees of Freedom ($df$)	Mean Square ($MS$)	$F$-Statistic	$p$-value
Regression (Model)	$\text{SSR}$	$k$	$\text{MSR} = \frac{\text{SSR}}{k}$	$F = \frac{\text{MSR}}{\text{MSE}}$	$P(F_{(k, n-k-1)} \ge F)$
Residual (Error)	$\text{SSE}$	$n - k - 1$	$\text{MSE} = \frac{\text{SSE}}{n - k - 1}$	—	—
Total	$\text{SST}$	$n - 1$	$\text{MST} = \frac{\text{SST}}{n - 1}$	—	—

Annotated Worked Example

Research purpose: Factors affecting infant birth weight ($n = 189$).

Outcome ($Y$): bwt (Infant birth weight in grams).

Predictors ($X$): Maternal age (age),
- maternal weight at last menstrual period (lwt),
- smoking status during pregnancy (smoke),
- history of hypertension (ht),
- uterine irritability (ui)

# Load MASS (pre-installed in R)
library(MASS)
data(birthwt)

# Convert binary indicator variables into explicit factors
birthwt$smoke <- factor(birthwt$smoke, labels = c("Non-Smoker", "Smoker"))
birthwt$ht    <- factor(birthwt$ht, labels = c("No", "Yes"))
birthwt$ui    <- factor(birthwt$ui, labels = c("No", "Yes"))

Regression results

model <- lm(bwt ~ age + lwt + smoke + ht + ui, data = birthwt)
summary(model)

Residuals:
     Min       1Q   Median       3Q      Max 
-1675.13  -464.38    -1.02   459.70  1779.69 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 2506.354    291.759   8.591 3.73e-15 ***
age            3.648      9.432   0.387  0.69935    
lwt            4.388      1.691   2.594  0.01024 *  
smokeSmoker -240.848    100.413  -2.399  0.01746 *  
htYes       -643.958    207.232  -3.107  0.00219 ** 
uiYes       -547.068    139.937  -3.909  0.00013 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 671.4 on 183 degrees of freedom
Multiple R-squared:  0.1747,    Adjusted R-squared:  0.1522 
F-statistic:  7.75 on 5 and 183 DF,  p-value: 1.234e-06

ANOVA table

library(olsrr)
olsaov<-ols_regress(model)$anova
olsaov$anova

                                 ANOVA                                   
------------------------------------------------------------------------
                    Sum of                                              
                   Squares         DF    Mean Square      F        Sig. 
------------------------------------------------------------------------
Regression    17469477.719          5    3493895.544     7.75    0.0000 
Residual      82500178.091        183     450820.645                    
Total         99969655.810        188                                   
------------------------------------------------------------------------

Key Metrics Calculations

$R^2$ Calculation: \[R^2 = \frac{17469477.719 }{ 99969655.810} = \quad (17.4\%)\] Interpretation: 12.6% of the variance in biomarker reduction is explained by maternal weight at last menstrual period, baseline metabolic index, and age.
Adjusted $R^2$ Calculation: \[R^2_{\text{adj}} = 1 - \left( \frac{450820.645}{99969655.810/ 188} \right) = = 1 - 0.8478 = 0.1522 \] Interpretation: After penalizing for adding three model parameters, the model accounts for approximately 70.2% of the variance.
$F$-Test Evaluation: $F_{(5, 183)} = 7.75, \quad p < 0.0001$$ Conclusion: Since the $p$-value is well below $\alpha = 0.05$, we reject $H_0$. The model as a whole explains a statistically significant amount of variation compared to a flat horizontal line.

Simple Linear Regression: $t$-test vs. $F$-test

In simple linear regression ($k = 1$): \[ F_{(1, n-2)} = (t_{\beta_1})^2 \]

The two-sided $t$-test on the single slope coefficient $\beta_1$ and the overall $F$-test are mathematically identical and yield the exact same $p$-value.

High $F$-statistic, but no individual significant $t$-statistics?

This classic paradox typically occurs due to severe multicollinearity.

The predictors are highly correlated with each other.
The overall $F$-test confirms that the set of predictors collectively accounts for substantial variation.
However, the variance inflation factor (VIF) is high, inflating individual standard errors and making it impossible for the individual $t$-tests to isolate which specific variable is doing the work.

Low $R^2$ with a Statistically Significant $F$-test

Common in noisy biological or social science data (e.g., $R^2 = 0.08$, $p < 0.001$ with $n = 5,000$).
This means the effect is genuine, real, and reproducible (not a sampling artifact), but individual-level variation remains high due to unmeasured factors.

Pitfalls to Avoid

Confusing $R^2_{\text{adj}}$ with Causality: A high $R^2$ indicates strong linear correlation, not causal direction.
Comparing $R^2_{\text{adj}}$ Across Different Transformations: You cannot compare the $R^2$ of a model predicting $Y$ directly to a model predicting $\ln(Y)$, because $\text{SST}$ is measured on fundamentally different scales.
Over-reliance on $R^2_{\text{adj}}$ Alone: Always inspect residual diagnostics (residual vs. fitted plots, Q-Q plots) to ensure linearity, normality, and homoscedasticity assumptions hold.

8. Summary Diagnostic Flowchart

[Fit OLS Regression Model]
           │
           ▼
[Check Overall F-Test]
  ├── p >= 0.05 ───► Model does not explain variance beyond noise. Stop / Respecify.
  └── p < 0.05 ────► Model explains significant variance. Proceed:
                           │
           ┌───────────────┴──────────────┐
           ▼                              ▼
    [Assess Fit]                  [Inspect Coefficients]
    • Check R² & Adj R²           • Examine t-tests & p-values
    • Check Root MSE (s_e)        • Check signs & magnitude
                                  • Check VIF for multicollinearity
                                          │
                                          ▼
                               [Validate Assumptions]
                               • Residual vs Fitted (Linearity / Homoscedasticity)
                               • Normal Q-Q Plot (Normality of residuals)
                               • Leverage / Cook's D (Influential outliers)

Eakins, Thomas. 1875. The Clinic of Dr. Gross. Oil on canvas. Philadelphia Museum of Art.

Source of Variation	Sum of Squares (\(SS\))	Degrees of Freedom (\(df\))	Mean Square (\(MS\))	\(F\)-Statistic	\(p\)-value
Regression (Model)	\(\text{SSR}\)	\(k\)	\(\text{MSR} = \frac{\text{SSR}}{k}\)	\(F = \frac{\text{MSR}}{\text{MSE}}\)	\(P(F_{(k, n-k-1)} \ge F)\)
Residual (Error)	\(\text{SSE}\)	\(n - k - 1\)	\(\text{MSE} = \frac{\text{SSE}}{n - k - 1}\)	—	—
Total	\(\text{SST}\)	\(n - 1\)	\(\text{MST} = \frac{\text{SST}}{n - 1}\)	—	—

Regression diagnostics

Learning objectives

Overview of linear regression

Evaluating linear regression - from an observation

A summary of error

Mathematical Formulations & Intuitions

Goodness-of-Fit: \(R^2\)

Adjusted \(R^2\) (\(R^2_{\text{adj}}\))

\(F\)-Test for Model Significance