The Modern Mathematics of Deep Learning

Pruning hypotheses on a finite network

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Pruning · Describe removal of network weights and related error bounds.

The project uses a linear rule with two features. The fitted weights produce scores or normalized mass.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 6.5 Removing connections

Check the source conditions for one ReLU network and count the parameters removed by a chosen mask.

This example makes the source notation or condition concrete.

Source formula example

\|w_{i}^{(1)}\|_{2}\leq 1/2

illustration · This specific constructed network satisfies the hypotheses and illustrates a 99% sparse parameter result; arbitrary mask choices need not give the source error bound.

Symbols

$n$: Original hidden-unit count, at least 100.
$wᵢ⁽¹⁾,bᵢ⁽¹⁾$: Incoming weight and bias of unit i.
$w⁽²⁾$: Output-weight vector.
$I$: Set of retained units.
$μ$: Probability measure supported on the unit ball.
$fᵢ$: Rescaled hidden feature.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Incoming weight

This example’s data flow

You choose

input.hiddenWeight = 0.5

Initial values in code

n = 100
bias = 0.5
keep = [0]

Calculated here

Result expression: { n, hiddenWeight, bias, hiddenNorm: Math.abs(hiddenWeight), weightBound: Math.abs(hiddenWeight) <= 0.5, biasBound: bias <= 0.5, keepCount: keep.length, keepBound: keep.length <= n / 100, zeroOutputWeights: keptOutputWe...

hiddenWeight = input.hiddenWeight Uses: input.hiddenWeight
originalOutputWeights = Array(n).fill(1) Uses: n
keptOutputWeights = originalOutputWeights.map((_, i) => (i === 0 ? 100 : 0)) Uses: originalOutputWeights
originalValue = originalOutputWeights.reduce( (s, w) => s + w * Math.max(0, bias), 0, ) Uses: originalOutputWeights, bias
prunedValue = keptOutputWeights.reduce((s, w) => s + w * Math.max(0, bias), 0) Uses: keptOutputWeights, bias
feature = 100 * Math.max(0, bias) Uses: bias

Returns

Example result

n: 100
hiddenWeight: 0.5
bias: 0.5
hiddenNorm: 0.5
weightBound: true
biasBound: true
keepCount: 1
keepBound: true

… (13 items)

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

The toy uses n=100 identical scalar hidden weights and biases, with μ concentrated at x=0. The input norm and bias bounds hold. All original output weights are 1, so their L1 norm is 100 and fᵢ(0)=50. Retaining one unit with output weight 100 reproduces the original output at x=0, while 99 of the output weights and all removed incoming parameters can be set to zero. The signed feature collection contains fᵢ and −fᵢ; if every output weight is zero the original network output is zero.

Source: Source expression