The Modern Mathematics of Deep Learning

Ridge regression

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

Classification predicts three species; Ridge predicts body mass in grams using the same observations. These are separate models. Training predictions feed the objective; 102 separate test predictions never update the weights.

Objectives · Combine losses and penalties into a criterion for parameters or rules.

Project predictions and known answers produce losses. Classification uses cross-entropy and α/2 times squared non-bias weights; Ridge uses normalized-mass MSE and α times all squared coefficients, including the intercept.

TaskTraining rowsTraining stepsPenalty αHeld-out row ID

Objective

Input: Mean training loss: ≈0.2556; Penalty: ≈0.0005264
Output: ≈0.2561

Project operation

The project calculation uses these records; the source formula and example follow below. Code edits run locally; project settings travel between pages.

Mean training loss → Objective → Gradient

Project input

Showing first three entries; the operation receives the complete array. Rounded preview; the operation uses full precision.

input.meanLoss · Mean training loss: ≈0.2556
input.weights · Model: [≈-2.422E-17, ≈0.2558, ≈0.679]
input.alpha · Project settings or constant: 0.001
input.task · Project settings or constant: ridge

JavaScript

Result

≈0.2561

Used in these source sections: 1.3 The questions ahead

The three invented training pairs give pack counts and known sticker counts: (1, 2), (2, 4), and (3, 6). In real applications, raw observations are collected and may be cleaned or turned into features; labels are known answers. Here the invented counts go directly into xs and ys without preprocessing.

With no intercept, the chosen weight is stickers per pack. Predict each count, square each error, and average the results. Add α times the squared weight.

The penalty favors smaller weights while the error measures fit.

Source formula example

\frac{1}{m}\sum_{i=1}^{m}(\Phi(X^{(i)},\theta)-Y^{(i)})^{2}+\alpha\|\theta\|_{2}^{2}

exact · The result is exact for these finite toy inputs, apart from ordinary computer rounding.

Symbols

$m$: Number of training examples.
$x^{(i)}, y^{(i)}$: Input and target of example i.
$θ$: Real-valued weight vector; this toy has one entry.
$α$: Nonnegative penalty strength.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Try weight 2. Which part of the score remains? Then set α to 0.

WeightPenalty α

This example’s data flow

You choose

input.weight = 1
input.alpha = 0.001

Initial values in code

xs = [1, 2, 3]
ys = [2, 4, 6]

Calculated here

Result expression: evaluateRidge(xs, ys, input.weight, input.alpha)

weight ← input.weight
alpha ← input.alpha

predictions = xs.map((x) => predict(x, weight)) Uses: xs, predict, weight
errors = predictions.map((prediction, i) => prediction - ys[i]) Uses: predictions, ys
squaredErrors = errors.map(squaredLoss) Uses: errors, squaredLoss
mse = mean(squaredErrors) Uses: mean, squaredErrors
penalty = alpha * weight ** 2 Uses: alpha, weight

Returns

Example result

xs: array (3 items): [1, 2, 3]
ys: array (3 items): [2, 4, 6]
predictions: array (3 items): [1, 2, 3]
errors: array (3 items): [-1, -2, -3]
squaredErrors: array (3 items): [1, 4, 9]
mse: ≈4.667
penalty: 0.001
objective: ≈4.668

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

This scores the reader-chosen weight; it does not fit a model or find the best weight. The reader also chooses α, whose default 0.001 comes from the source figure. An optimizer could use the objective to try another weight; a fitted weight could then predict a sticker count from a new pack count.

Source: Figure 1.4 · paper pp. 18