The Modern Mathematics of Deep Learning

Sparse approximation bound

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Pruning · Describe removal of network weights and related error bounds.

The project uses a linear rule with two features. The fitted weights produce scores or normalized mass.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 6.5 Removing connections

Compare a finite approximation error with both bounds in the source pruning argument.

These operations define structured network layers and their analysis.

Source formula example

\left\|\Phi_{a}(\cdot,\theta)-\sum_{i\in I}c_{i}f_{i}\right\|_{L^{2}(\mu)}\leq\frac{\|w^{(2)}\|_{1}}{\sqrt{|I|}}\leq\frac{\sqrt{200}\|w^{(2)}\|_{1}}{\sqrt{n}},

illustration · This special zero-error example does not construct the source’s sparse approximation in general or establish the existence result.

Symbols

$I$: Selected nonempty set of hidden-unit indices.
$fᵢ$: Rescaled hidden feature i.
$cᵢ$: Real coefficients.
$μ$: Probability measure in the source assumptions.
$\Vert w⁽²⁾\Vert ₁$: Sum of absolute output weights.
$n$: Number of original hidden units.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Original unitsOutput weight L1 norm

This example’s data flow

You choose

input.n = 100
input.norm = 2

Initial values in code

networkValue = 0
approximation = 0

Calculated here

Result expression: { count, countLowerBound: n / 200, countBoundHolds: count >= n / 200, error, firstBound, secondBound, }

n = input.n Uses: input.n
norm = input.norm Uses: input.norm
count = Math.floor(n / 100) Uses: n
error = Math.abs(networkValue - approximation) Uses: networkValue, approximation
firstBound = norm / Math.sqrt(count) Uses: norm, count
secondBound = (Math.sqrt(200) * norm) / Math.sqrt(n) Uses: norm, n

Returns

Example result

count: 1
error: 0
firstBound: 2
secondBound: ≈2.828
countLowerBound: 0.5
countBoundHolds: true

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

The source chooses |I| = floor(n/100), which is at least n/200 for n ≥ 100. The toy evaluates the pointwise error for zero input with zero biases and an L² measure concentrated there, so the error is zero, then evaluates both right-hand bounds. The source obtains its coefficients using the convex-hull lemma; arbitrary choices do not automatically satisfy the bound.

Source: Source expression