The Modern Mathematics of Deep Learning

Effective dimension and variance

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Generalization and risk bounds · Relate sample behavior to population risk under stated assumptions.

The project supplies finite observations and fitted parameters. A held-out sample does not establish population bounds or prove a theorem.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 2.4 A second descent

Sort covariance eigenvalues from largest to smallest. For each k, divide the remaining eigenvalue sum by the next eigenvalue and find the first k meeting the threshold.

The source then uses k* in an approximate variance expression.

The spectrum can affect generalization beyond the raw count of parameters.

Source formula example

k^{*}:=\min\{k\geq 0\colon\frac{\sum_{i>k}\lambda_{i}}{\lambda_{k+1}}\geq cm\},

approximation · The output evaluates the source approximation using supplied eigenvalues and c. It does not compute actual variance or specify the constants hidden in the approximate relation.

Symbols

$λ_i$: Nonnegative eigenvalues of the input covariance in decreasing order.
$c$: A positive universal constant in the source condition.
$m$: Positive sample size.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Constant cSample size m

This example’s data flow

You choose

input.c = 1
input.m = 2

Initial values in code

eigenvalues = [3, 2, 1]

Calculated here

Result expression: { k, tailSum, tailSquares, approximation: k / m + (m * tailSquares) / tailSum ** 2, }

c = input.c Uses: input.c
m = input.m Uses: input.m
k = eigenvalues.findIndex( (value, index) => eigenvalues.slice(index).reduce((a, b) => a + b, 0) / value >= c * m, ) Uses: eigenvalues, c, m
tail = eigenvalues.slice(k) Uses: eigenvalues, k
tailSum = tail.reduce((a, b) => a + b, 0) Uses: tail
tailSquares = tail.reduce((sum, value) => sum + value * value, 0) Uses: tail

Returns

Example result

k: 0
tailSum: 6
tailSquares: 14
approximation: ≈0.7778

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

The source considers normally distributed inputs with covariance rank greater than m. Under the stated condition that k*/m is sufficiently small, it gives the variance expression with high probability up to the approximate relation. The code uses three positive eigenvalues, rejects a missing k, and does not determine the theorem constant or verify high probability.

Source: Source expression