Linear regression model is formally defined as \[
\begin{aligned}
y_{i} = & \beta_{0} + \beta_{1}x_{1i} + \beta_{2}x_{2i} + \dots + \beta_{p}x_{pi} + \epsilon_{i} \text{, or}\\
\mathbb{y} = &\mathbb{\beta}\mathbb{X}+ \mathbb{\epsilon}
\end{aligned}
\]
Systematic Part (Independent variable, predictor)
Random Part: Individual observations of the outcome vary by an error term \(\epsilon_{i}\).
It is assumed that \(\epsilon_{i} \sim \mathcal{N}(0, \sigma_{\epsilon}^{2})\)
Prediction in regression
Confidence Interval (CI) for the Mean Response:
Definition: the average expected outcome for all individuals with a specific predictor value \(x\)
Uncertainty: Estimated regression line
Prediction Interval (PI) for a New Observation:
Definition: Predict the specific outcome \(y\) for a single, new individual with a specific predictor value \(x\).
Uncertainty: Estimated regression line, and and random scatter of individual data points around that line
Evaluating regression
\(y_i\): The actual observed value.
\(\hat{y}_i\): The predicted value on the regression line.
\(\bar{y}\): The mean of all observed \(y\) values.
The following relationship holds \[TSS = ESS + RSS\] TSS (Total Sum of Squares): \[\sum (y_i - \bar{y})^2\] ESS (Explained Sum of Squares): \[\sum (\hat{y}_i - \bar{y})^2\] RSS (Residual Sum of Squares):\[\sum (y_i - \hat{y}_i)^2\]
\(R^2\): The Coefficient of Determination
\(R^2\) is the percentage of variance explaine dby the linear model
\[
R^2 = \frac{ESS}{TSS} = 1 - \frac{RSS}{TSS}
\]
-\(R^2 = 0\): the model explains none of the variance in the outcome (the regression line is flat at the mean, \(\bar{y}\)). - \(R^2 = 1\): the model explains 100% of the variance (every data point falls exactly on the regression line, meaning RSS is zero).
Inference on linear model
Even we have \(R^2\), we need to know if explained variance s statistically significant.
Implication in linear model: Including other predictors to control for confounding commonly has the effect of difficult to reject the null hypothesis
Some of the unadjusted association is explained by those other predictors.
Assumption on linear regression
Linearity
Normality of erros
Independece of errors
Homoscedasticity
No multicollinearity
Linearity
The model assumes that the average value of the outcome changes linearly with each continuous predictor. Departures from linearity can be checked using Residual versus Predictor (RVP) plots or Component-Plus-Residual (CPR) plots. Applying a nonparametric smoother (like LOWESS) to these plots helps detect nonlinear trends in the residuals. If nonlinearity is found, you may need to apply transformations to the predictor (e.g., adding quadratic terms, log transformations) or categorize the continuous predictor.
Normality of Errors
The error term \(\epsilon\) is assumed to have a normal distribution. This can be assessed using histograms, boxplots, kernel density estimates, and normal quantile-quantile (Q-Q) plots applied to the residuals. In a normal Q-Q plot, upward curvature is diagnostic of right-skewness. Transforming the outcome variable (e.g., replacing \(y\) with \(log(y)\)) is often successful for reducing the skewness of residuals.
Constant Variance (Homoscedasticity)
The model assumes that the variance of the error term, \(\sigma_{\epsilon}^{2}\), is constant across all observations. Violations of this assumption are known as heteroscedasticity, which can affect the validity of confidence intervals and P-values. Homoscedasticity is evaluated using RVP plots and Residual versus Fitted (RVF) plots. A horizontal funnel shape in these scatterplots is a signal of heteroscedasticity.
Influential Points and Outliers
High leverage points are x-outliers with the potential to exert undue influence on regression coefficient estimates. Influential points are specific observations that have actually exerted an undue influence on the estimates. DFBETA statistics are recommended to quantify how much each coefficient would change if a specific observation were omitted from the dataset. IV. Code Illustrations: Python and RBelow are examples of how to fit a multiple linear regression model and generate the necessary diagnostic plots in both Python
# 1. Generate Synthetic Dataset.seed(42)n <- 200X1 <- rnorm(n, mean=50, sd=10)X2 <- rnorm(n, mean=100, sd=20)# Create outcome with normal noiseY <- 15 + 2.5*X1 - 1.2*X2 + rnorm(n, mean=0, sd=15)data <- data.frame(Y, X1, X2)# 2. Fit the Linear Regression Modelmodel <- lm(Y ~ X1 + X2, data=data)# Print model summary (Contains F-test, RSS, and coefficients)print(summary(model))# 3. Residual Diagnostics# R has built-in diagnostic plots that are generated by plotting the model objectpar(mfrow=c(2,2)) # Set up a 2x2 grid for plots# Plot 1: Residuals vs Fitted (checks linearity and constant variance)# Plot 2: Normal Q-Q (checks normality of residuals)# Plot 3: Scale-Location (checks homoscedasticity)# Plot 4: Residuals vs Leverage (checks for influential points / high leverage)plot(model)
