Linear Regression (LR) Models

• is a type of continuous regression model whose function/estimator is linear with respect to the regression coefficients {𝜃₀, ..., 𝜃_𝑝}:
• 𝑦̂ = 𝜃₀ + 𝜃₁𝑓₁(𝒙) + ... + 𝜃_𝑝𝑓_𝑝(𝒙)
• models the relationship between:
  • 𝑌 - a single scalar response/dependent variable (for categorical use logistic regression)
  • {𝑋₁, ..., 𝑋_𝑘} - one or more regressors or explanatory/predictor/covariate/independent variables. predictor variable types:
    • continuous/scalar/numerical predictor
    • discrete/categorical predictor - itself can be either nominal or ordinal
• models mean/expected response as a function/conditional of regressors (where 𝑓_𝑖(..) are feature functions)
  • 𝐄[𝑌|𝑋₁=𝑥₁, ..., 𝑋_𝑘=𝑥_𝑘] = ℎ(𝑥₁, ..., 𝑥_𝑘) = 𝑦̂ = 𝜃₀ + 𝜃₁𝑓₁(𝑥₁, ..., 𝑥_𝑘) + ... + 𝜃_𝑝𝑓_𝑝(𝑥₁, ..., 𝑥_𝑘)
  • coefficient 𝜃₀ represents the 𝑦 intercept when all feature functions 𝑓_𝑖(..) equate to 0
  • coefficient 𝜃_𝑖 represents the mean change in the dependent variable 𝑦 given a 1 unit change in the independent feature function 𝑓_𝑖(𝑥₁, ..., 𝑥_𝑘) # for 1≤𝑖≤𝑝
• is a type of level-level model (or even a level-log model when 𝑓_𝑖(..) are log functions)
• the dependent variable 𝑦 is the combination of the regression model and error
  • 𝑦 = 𝑦̂ + 𝑒
  • dependent variable = (constant + independent variables) + error
  • dependent variable = deterministic + stochastic
  • deterministic component is the portion of the variation in the dependent variable that the independent variables explain. In other words, the mean of the dependent variable is a function of the independent variables. In a regression model, all of the explanatory power should reside here
  • error is the difference between the expected value 𝑦̂ and the observed value 𝑦. Let’s put these terms together—the gap between the expected and observed values must not be predictable. Or, no explanatory power should be in the error. If you can use the error to make predictions about the response, your model has a problem. This issue is where residual plots play a role.
  • the theory here is that the deterministic component of a regression model does such a great job of explaining the dependent variable that it leaves only the intrinsically inexplicable portion of your study area for the error. If you can identify non-randomness in the error term, your independent variables are not explaining everything that they can

LR - Steps

given sample/training data:

• (𝑦₁, 𝑥₁₁, ..., 𝑥_1𝑘) # sample 1
• (𝑦₂, 𝑥₂₁, ..., 𝑥_2𝑘) # sample 2
• ...
• (𝑦_𝑛, 𝑥_𝑛1, ..., 𝑥_𝑛𝑘) # sample 𝑛

the task of Linear Regression:

• choose line equation form, such as:
  • 𝐄[𝑌|𝑋₁=𝑥₁] = 𝑦̂ = ℎ(𝑥₁) = 𝜃₀ + 𝜃₁𝑥₁ # univariate linear regression

  • 𝐄[𝑌|𝑋₁=𝑥₁, 𝑋₂=𝑥₂] = 𝑦̂ = ℎ(𝑥₁,𝑥₂) = 𝜃₀ + 𝜃₁𝑥₁ + 𝜃₂𝑥₂ # multivariate linear regression
    

  • 𝐄[𝑌|𝑋₁=𝑥₁, 𝑋₂=𝑥₂] = 𝑦̂ = ℎ(𝑥₁,𝑥₂) = 𝜃₀ + 𝜃₁𝑥₁𝑥₂ + 𝜃₂𝑥₁² + 𝜃₃𝑥₂ # multiple linear regression
• where:
  • 𝐄[𝑌|..] and 𝑦̂ and ℎ(..) - scalar response/dependent variable or hypothesis function conditional on 𝑥_𝑖's
  • 𝑥_𝑖 - regressors or explanatory/predictor/covariate/independent variables
    
  • 𝜃_𝑖 - regression coefficients/weights
• estimate/find the values of the regression coefficients 𝜃_𝑖 which best fit the line equation to the data
• determine whether its a goodfit

LR - Types

LR - Methods for Estimating Coefficients (𝜃_𝑖)

Methods estimating unknown coefficients {𝜃₀, ..., 𝜃_𝑘} of 𝐄[𝑌|𝑋₁=𝑥₁, ..., 𝑋_𝑘=𝑥_𝑘] = ℎ(𝑥₁, ..., 𝑥_𝑘) = 𝑦̂ = 𝜃₀ + 𝜃₁𝑓₁(𝑥₁, ..., 𝑥_𝑘) + ... + 𝜃_𝑘𝑓_𝑘(𝑥₁, ..., 𝑥_𝑘)

LR - Model Types

Linear Regression Models - takes an input vector 𝑥∊ℝ^𝑛 as input and predicts the value of a scalar 𝑦∊ℝ as output (whose function/estimator is linear wrt the regression coefficients {𝜃₀, ..., 𝜃_𝑝})

LR - Methods for Determining How Well The Fitted Line Describes the Data

• Model - Performance/Accuracy/Evaluation/Goodness-of-Fit Measures/Metrics/Analysis
  

LR - Methods for Diagnosing Bias Variance 

• ML - Diagnosing Model Bias/Underfit vs Variance/Overfit - Linear Regression

• LR - Model Building

LR - Subpages

• Linear Regression vs Gaussian Regression
  

• Bayesian Linear Regression
• Cook's Distance
• Elastic Net Regression (Ridge & LASSO)
• LASSO Regression (Least Absolute Shrinkage and Selection Operator)
• Linear Regression (LR) Models - Comparisons
• LR - ANOVA Table
• LR - Categorical Predictor Variables
• LR - Methods Estimating Unknown Regression Coefficients
• LR - Model Building
• LR - Problems
• LR - R Code Examples
• LR - Standard Regression Assumptions
• LR - Tests - Derivation of F-Statistic
• LR - Tests - Derivation of Student T-Statistic
• Mallow's Cp Statistic
• Multivariate/Multiple Linear Regression Models
• Ordinary Least Squares (OLS) Regression
• Ridge Regression
• Univariate/Single-Variable/Simple Linear Regression Models

LR - Resources

• Zed Statistics Regression Playlist
• StatQuest Linear Models: Part 1 & Part 2