Linear Models & Regularization

A linear model is a weighted sum

For a feature vector x, a linear regression model predicts ŷ = wᵀx + b. Each coefficient describes the model's change in prediction for a one-unit increase in that feature while other features are held constant. Ordinary least squares chooses coefficients that minimize squared residuals. Linear models can use many features, but they impose a linear relationship in the represented feature space.

from sklearn.linear_model import LinearRegression

model = Pipeline([
    ("prepare", preprocess),
    ("regressor", LinearRegression()),
])
model.fit(X_train, y_train)
predictions = model.predict(X_valid)

Regularization penalizes large coefficients

Ridge adds an L2 penalty proportional to the sum of squared coefficients. It shrinks coefficients toward zero but usually does not set them exactly to zero. Lasso uses an L1 penalty and can produce exact zero coefficients. Elastic Net combines both. The strength parameter controls the trade-off between fitting training data and keeping coefficients small. Scale numeric features before comparing penalties; otherwise the penalty treats units unevenly. Tune the strength using validation or cross-validation, never the final test set.