The Modern Mathematics of Deep Learning

A bounded squared loss

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Individual losses · Give one observed example a loss score.

Project predictions and known answers produce losses. Classification uses cross-entropy and α/2 times squared non-bias weights; Ridge uses normalized-mass MSE and α times all squared coefficients, including the intercept.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 1.2.3 Does it work on new examples?

If both a regression target and every prediction have absolute value at most c, their difference has absolute value at most 2c. Squaring gives the loss bound 4c².

The toy class contains the two constant predictions −c and +c, and the target is +c.

Bounded losses enable the source’s concentration arguments.

Source formula example

\sup_{f\in\mathcal F}|f(X)|\leq c,\quad\sup_{f\in\mathcal F}|\mathcal L(f,Z)|\leq4c^2

illustration · The maxima are exact for this two-rule toy class. The general source assumptions concern almost-sure bounds over the full class and distribution.

Symbols

$c$: A positive bound on |Y| and on |f(X)| across the hypothesis class.
$ℒ$: Squared regression loss.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Positive bound c

This example’s data flow

You choose

input.c = 2

Calculated here

Result expression: { predictions, target, losses, largestPredictionMagnitude: Math.max(...predictions.map(Math.abs)), largestLoss: Math.max(...losses), lossBound: 4 * c ** 2, }

c = input.c Uses: input.c
predictions = [-c, c] Uses: c
target = c Uses: c
losses = predictions.map((prediction) => (prediction - target) ** 2) Uses: predictions, target

Returns

Example result

predictions: array (2 items): [-2, 2]
target: 2
losses: array (2 items): [16, 0]
largestPredictionMagnitude: 2
largestLoss: 16
lossBound: 16

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

The source assumes |Y|≤c and sup_f |f(X)|≤c almost surely. The triangle inequality gives |f(X)−Y|≤2c and hence squared loss at most 4c². The calculation does not estimate a population supremum from samples.

Source: Bounded regression loss