The Modern Mathematics of Deep Learning

Gaussian initialization density

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Initialization · Describe initial parameters and their distributions.

The current objective supplies a gradient; training updates the model parameters.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 5.2 When parameters move only a little

The source initializes each parameter independently with a mean-zero Gaussian whose variance is 1/n.

The distribution describes random starting weights for the training analysis.

Source formula example

\Theta^{(1)}_{j}\sim{\mathcal{N}}(0,1/n)^{d+1}

illustration · This evaluates a density, not a random draw or a proof of convergence.

Symbols

$Θ⁽¹⁾ⱼ$: Incoming weights and bias of hidden unit j, a vector in ℝᵈ⁺¹.
$Θ⁽²⁾ⱼ$: Scalar output weight of hidden unit j.
$N(0,1/n)$: Normal distribution with mean zero and variance 1/n.
$n$: Positive number of hidden units.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Hidden units

This example’s data flow

You choose

input.n = 4

Initial values in code

coordinates = [0, 0, 0]

Calculated here

Result expression: { variance, coordinates, oneCoordinateDensity: densityAtZero, jointDensity: densityAtZero ** coordinates.length, }

n = input.n Uses: input.n
variance = 1 / n Uses: n
densityAtZero = 1 / Math.sqrt(2 * Math.PI * variance) Uses: variance

Returns

Example result

variance: 0.25
coordinates: array (3 items): [0, 0, 0]
oneCoordinateDensity: ≈0.7979
jointDensity: ≈0.5079

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

For d = 1 the three coordinates represent two incoming entries and one output weight at zero. Their joint density is the product because initialization coordinates are independent. Density is not a probability at a single point.

Source: Source expression