The Modern Mathematics of Deep Learningexplained for non-mathematicians

Concepts / Optimization

Parameters along a low-loss construction

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Optimization theory · Study convergence and loss geometry under assumptions.

The project supplies finite observations and fitted parameters. A held-out sample does not establish population bounds or prove a theorem.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 5.1 The shape of training loss

Duplicate hidden features allow output weights to be combined, leaving zero weights on units whose incoming parameters may then change.

This calculation explains the corresponding source expression.

Source formula example

(θ∗(1),θ(2))(\theta_{*}^{(1)},\theta^{(2)})

illustration · The example demonstrates the parameter pairs and exact equalities, not the source probabilistic covering or path-loss guarantee.

Symbols

θ∗(1)θ*⁽¹⁾
Hidden parameters after duplicating features.
θ(2)θ⁽²⁾
Original output weights.
θ∗(2)θ*⁽²⁾
Output weights with the latter half zero.
θ^(1)θ̂⁽¹⁾
Replacement hidden parameters.
ΦΦ
Network function.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

This example’s data flow

You choose

  • input.x = 1

Initial values in code

  • hidden = [1, 2, 1, 2]
  • originalOutputWeights = [1, 2, 3, 4]
  • combinedOutputWeights = [4, 6, 0, 0]
  • replacementHidden = [1, 2, 3, 4]

Calculated here

Result expression: { width, halfWidth: width / 2, evenWidth: width % 2 === 0, hidden, originalOutputWeights, combinedOutputWeights, replacementHidden, before: network(hidden, originalOutputWeights), combined: network(hidden, combinedOutpu...

  • x = input.x Uses: input.x
  • width = hidden.length Uses: hidden

Returns

Example result
hidden
array (4 items): [1, 2, 1, 2]
originalOutputWeights
array (4 items): [1, 2, 3, 4]
combinedOutputWeights
array (4 items): [4, 6, 0, 0]
replacementHidden
array (4 items): [1, 2, 3, 4]
before
16
combined
16
replaced
16
width
4
… (10 items)

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

The toy has four units: unit 3 duplicates 1 and unit 4 duplicates 2. Combining output weights gives [4,6,0,0]. Thus replacing hidden weights of the last two units leaves the function unchanged. The source uses a nearest-neighbor replacement, local Lipschitz risk, bounded output-weight L1 norm and sufficiently large even width; this toy starts with exact duplicates.

Source: Source expression