LASSO and RIDGE linear regression – better prediction or much more?

Siniša Opić 

University of Zagreb, Faculty of Teacher Education

sinisa.opic@ufzg.hr

Education for innovation and research

Abstract

The paper examines the use, limitations, and prerequisites for implementing LASSO (Least Absolute Shrinkage and Selection Operator) and RIDGE linear regression techniques. These techniques aim to improve the traditional regression method by reducing the sum of squared errors between observed and predicted values (OLS). LASSO and RIDGE introduce a penalty term to the regression equation to address overfitting. These two techniques differ in their penalty term (L1 for LASSO and L2 for RIDGE,) and their impact on the coefficients (RIDGE tends to reduce coefficients towards zero but not entirely, while LASSO can force some coefficients to be exactly zero. In a certain empirical example, the results of the regression line of linear regression with RIDGE and LASSO are analyzed and compared. The paper presents the issue of graphical overfitting in classical regression and its resolution using the mentioned regression procedures to reduce variance. The paper also examines the L1 and L2 regularization (cost function) procedures and their role in solving the problem of predictor multicollinearity in the example. Hybrid elastic Net regression that combines both L1 (LASSO) and L2 (RIDGE) regularization in cases when λ1>0, and λ2>0, are also discussed.

Key words