The Modern Mathematics of Deep Learning

The printed bias–variance equality

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Generalization and risk bounds · Relate sample behavior to population risk under stated assumptions.

The project supplies finite observations and fitted parameters. A held-out sample does not establish population bounds or prove a theorem.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 2.4 A second descent

The paper prints the L² norm without a square in this chain. A simple one-dimensional example gives different values for the printed middle and right-hand quantities.

Let X be uniformly ±1, let the true parameter be zero, and let independent centered noise be uniformly ±a. With one training sample, the fitted slope is noise divided by X.

The example checks the printed equality using fully specified random variables.

Source formula example

\begin{split}{\mathbbm{E}}[{\mathcal{R}}(\widehat{f}_{S})|(X^{(i)})_{i=1}^{m}]-{\mathcal{R}}^{*}&={\mathbbm{E}}[\|\widehat{f}_{S}-f^{*}\|_{L^{2}({\mathbbm{P}}_{X})}|(X^{(i)})_{i=1}^{m}]\\ &=(\theta^{*})^{T}P{\mathbbm{E}}[XX^{T}]P\theta^{*}+{\mathbbm{E}}[\nu^{2}]{\operatorname{Tr}}\big(\Sigma^{+}{\mathbbm{E}}[XX^{T}]\big),\end{split}

exact · This checks a specific case of the displayed expression and does not determine the intended correction or disprove bias–variance decompositions in general.

Symbols

$X$: A centered real input, uniformly ±1 in this example.
$ν$: Independent centered noise, uniformly ±a.
$\hat f_S$: Minimum-norm empirical least-squares predictor.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Noise amplitude a

This example’s data flow

You choose

input.amplitude = 2

Initial values in code

bias = 0

Calculated here

Result expression: { conditionalExpectedNorm, excessRisk, bias, variance, sameAsPrintedMiddle: conditionalExpectedNorm === excessRisk, }

a = input.amplitude Uses: input.amplitude
conditionalExpectedNorm = a Uses: a
excessRisk = a * a Uses: a
variance = a * a Uses: a

Returns

Example result

conditionalExpectedNorm: 2
excessRisk: 4
bias: 0
variance: 4
sameAsPrintedMiddle: false

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

Take d=m=1, true parameter zero, X uniform ±1, and noise uniform ±a independent of X. The single sample matrix and its pseudoinverse are both 1, so the projection onto its kernel is zero. The fitted slope has absolute value a, hence its L² prediction norm is a; its excess squared risk and the variance term are a². For a=2 these values are 2 and 4. This calculation addresses the expression as printed; it does not assume what the authors intended.

Source: Printed bias–variance expression