The Modern Mathematics of Deep Learning

Training starts at initialization

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Learning and parameter updates · Select a rule or update parameters using a learning rule.

The current objective supplies a gradient; training updates the model parameters.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 5.2 When parameters move only a little

The initial training iterate equals the chosen initial parameter vector.

This example explains the corresponding source notation.

Source formula example

\Theta^{(0)}=\Theta

illustration · Specifying initialization and batch size does not establish convergence.

Symbols

$Θ$: Initial parameter vector.
$Θ⁽⁰⁾$: Iteration-zero parameter vector.
$m$: Number of training examples.
$m^{\prime}$: Batch size.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

First parameter

This example’s data flow

You choose

input.a = 1

Initial values in code

batchSize = 2
sampleCount = 2

Calculated here

Result expression: { initialization, iterationZero, batchSize, sampleCount, fullBatch: batchSize === sampleCount, }

initialization = [input.a, 2] Uses: input.a
iterationZero = [...initialization] Uses: initialization

Returns

Example result

initialization: array (2 items): [1, 2]
iterationZero: array (2 items): [1, 2]
batchSize: 2
sampleCount: 2
fullBatch: true

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

This toy copies initialization into the first iterate and uses all m=2 examples as the batch. The source discusses full-batch gradient descent with m′=m.

Source: Source expression