In linear regression, our fundamental objective is to explain the variability observed in the outcome variable (\(Y\)) using one or more explanatory variables (\(X_1, X_2, \dots, X_k\)).
Unexplained variation; sum of squared discrepancies between observed points and regression surface.
Goodness-of-Fit: \(R^2\)
\(R^2\) (The coefficient of determination) quantifies the proportion of total variation in the response variable explained by the linear combination of predictors:
Range: In standard OLS with an intercept, \(0 \le R^2 \le 1\) (or \(0\%\) to \(100\%\)).
Interpretation: If \(R^2 = 0.74\), exactly 74% of the variability in \(Y\) is explained by the predictors included in the model, while the remaining 26% is attributable to residual noise or omitted factors.
Limitation:\(R^2\) is non-decreasing with respect to the addition of predictors. Adding completely unrelated random noise variables will force \(\text{SSE}\) to stay equal or decrease, artificially inflating \(R^2\).
Adjusted \(R^2\) (\(R^2_{\text{adj}}\))
The Adjusted \(R^2\) incorporates degrees of freedom to penalize excessive parameterization and prevent overfitting:
ANOVA ------------------------------------------------------------------------ Sum of Squares DF Mean Square F Sig. ------------------------------------------------------------------------Regression 17469477.719 5 3493895.544 7.75 0.0000 Residual 82500178.091 183 450820.645 Total 99969655.810 188 ------------------------------------------------------------------------
Interpretation: 17.4% of the variance in infant birth weight is explained by maternal age, maternal weight at last menstrual period, smoking status, history of hypertension, and uterine irritability.
Adjusted \(R^2\) Calculation:\[
R^2_{\text{adj}} = 1 - \left( \frac{450820.645}{99969655.810/ 188} \right) = 1 - 0.8478 = 15.2\%
\]Interpretation: After penalizing for adding the model parameters, the model accounts for approximately 15.2% of the variance.
Common issues
Significant \(F\)-statistic, but no individual significant \(t\)-statistics
This classic paradox typically occurs due to severe multicollinearity.
The predictors are highly correlated with each other.
The overall \(F\)-test confirms that the set of predictors collectively accounts for substantial variation.
Low \(R^2\) with a Statistically Significant \(F\)-test
Common in noisy biological or social science data (e.g., \(R^2 = 0.08\), \(p < 0.001\) with \(n = 5,000\)).
This means the effect is genuine, real, and reproducible (not a sampling artifact), but individual-level variation remains high due to unmeasured factors.
L.I.N.E Assumptions
Overview
Letter
Assumption
Primary Diagnostic Tool
L
Linearity of the conditional expectation
\(\mathbb{E}[\varepsilon_i \mid X] = 0\)
I
Independence of observational errors
\(\text{Cov}(\varepsilon_i, \varepsilon_j) = 0 \quad (\forall i \neq j)\)
Confusing \(R^2_{\text{adj}}\) with Causality: A high \(R^2\) indicates strong linear correlation, not causal direction.
Comparing \(R^2_{\text{adj}}\) Across Different Transformations: You cannot compare the \(R^2\) of a model predicting \(Y\) directly to a model predicting \(\ln(Y)\), because \(\text{SST}\) is measured on fundamentally different scales.
Over-reliance on \(R^2_{\text{adj}}\) Alone: Always inspect residual diagnostics (residual vs. fitted plots, Q-Q plots) to ensure linearity, normality, and homoscedasticity assumptions hold.
Healthy Pattern: Residuals bounce randomly across a horizontal band centered at \(y = 0\), with no distinct trend. The red smooth smoother curve is nearly flat along \(y = 0\).
Violation Warning: Distinct parabolic “U-shape”, inverted “U”, or sinusoidal wave.
I — Independence of Errors
Knowing that Patient A has an unexpectedly high outcome provides zero information about Patient B’s outcome.
Example
Clustering: Patients recruited from the same hospital ward or family share correlated responses.
Autocorrelation: Longitudinal measurements taken repeatedly over time on the same subject.
Actions
Revise study design
Clustered data: Linear Mixed-Effects Models (lme4::lmer()) incorporating random intercepts for clusters/centers.
Tool: Evaluation of study design, Residuals vs. Time/Subject ID order, or Durbin-Watson test (car::dwt(model)).
car::dwt(model)
lag Autocorrelation D-W Statistic p-value
1 0.7582327 0.4817912 0
Alternative hypothesis: rho != 0
Consequence
Deceptive Precision: The effective sample size is smaller than \(n\). Linear regression underestimate standard errors (\(SE(\hat{\beta})\)), causing artificially narrow 95% CIs.
N — Normality of Residuals
The error terms are normally distributed around the fitted line.
Crucial Clarification: In regression, the raw variables \(Y\) and \(X\)do not need to be normally distributed; only the residuals (\(\varepsilon_i\)) must be normally distributed.
Tool:Normal Q-Q Plot (Base R Plot 2: plot(model, which = 2)).
Healthy Pattern: Residual quantiles fall tightly along the theoretical 45-degree dashed line.
Heavy tails (S-shape): Data has far more extreme clinical outliers than predicted by a normal curve.
Curvature at ends: Skewed distributions (frequently seen with right-skewed lab markers such as serum creatinine, viral loads, or cytokine concentrations).
E — Equal Variance
The variance of the residuals \(\text{Var}(\varepsilon_i \mid X) = \sigma^2\) is constant across all predicted values \(\hat{Y}\). (Homoscedasticity)
Consequence
Conventional \(SE(\hat{\beta})\) calculations are invalid. If variance expands at higher values, standard errors are underestimated for higher exposures, leading to false claims of significance.
Actions
Variance-stabilizing transformations:\(\log(Y)\) or \(\sqrt{Y}\).
Heteroscedasticity-Consistent (HC) Robust Standard Errors: Using sandwich::vcovHC(model, type = "HC3") paired with lmtest::coeftest().
Weighted Least Squares (WLS): Weights inversely proportional to variance: \(w_i = 1/\sigma_i^2\).
Tool:Scale-Location Plot (Base R Plot 3: plot(model, which = 3))
plot(model, which =3)
Healthy Pattern: A horizontal smooth red line with an even, constant vertical scatter of points across all fitted values.
Violation Warning: A megaphone / funnel shape where residual spread widens drastically as \(\hat{Y}\) increases (common in biomedical dosing and growth studies).
Summary Diagnostic Flowchart
[Fit Linear Model: lm(Y ~ X1 + X2)] │ ┌──────────────────────┴──────────────────────┐ ▼ ▼ [Evaluate Goodness-of-Fit] [Validate L.I.N.E.] • R² & Adjusted R² • Residuals vs Fitted (Linearity) • Residual Std Error (RSE) • Study structure (Independence) • Global ANOVA F-Test • Normal Q-Q (Normality) │ • Scale-Location (Equal Variance) │ │ └──────────────────────┬──────────────────────┘ │ All L.I.N.E. Assumptions Hold? │ ├─── YES ──► Report slopes, 95% CIs & p-values │ └─── NO ──► Apply Diagnostic Remedy: • Non-linear? ──► Log(X), Polynomials • Heteroscedastic? ──► Log(Y), HC3 Robust SE • Non-normal? ──► Transform Y, Bootstrap • Clustered? ──► Mixed-Effects Model (lmer)
Eakins, Thomas. 1875. The Clinic of Dr. Gross. Oil on canvas. Philadelphia Museum of Art.