The Modern Mathematics of Deep Learning

A sum of ridge functions

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Prediction rules and layers · Describe or apply prediction rules, layers and parameters.

The project uses a linear rule with two features. The fitted weights produce scores or normalized mass.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 3.1 A rule based on distance

Each term applies a one-variable function to a dot product. A suitable rotation can put the direction wⱼ in the first coordinate, so the same term depends only on that coordinate.

The toy uses two unit directions in two dimensions and ReLU for both terms.

This representation is used in the source Fourier argument for shallow networks.

Source formula example

\Phi=\sum_{j=1}^{n}\varrho_{j}(\langle w_{j},\cdot\rangle)=\sum_{j=1}^{n}(\varrho_{j}\otimes\mathds{1}_{{\mathbb{R}}^{d-1}})\circ R_{w_{j}}.

illustration · The toy uses two explicitly chosen rotations and real values. It does not establish the Fourier-support result for general networks.

Symbols

$w_j$: A unit direction in ℝᵈ.
$ρ_j$: A one-variable function associated with hidden unit j.
$R_{w_j}$: A rotation carrying wⱼ to the first coordinate direction.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

First coordinateSecond coordinate

This example’s data flow

You choose

input.x1 = 1
input.x2 = 2

Calculated here

Result expression: { directTerms, rotatedTerms, directSum: directTerms[0] + directTerms[1], rotatedSum: rotatedTerms[0] + rotatedTerms[1], }

x = [input.x1, input.x2] Uses: input.x1, input.x2
directTerms = [Math.max(0, x[0]), Math.max(0, x[1])] Uses: x
rotated1 = [x[0], x[1]] Uses: x
rotated2 = [x[1], -x[0]] Uses: x
rotatedTerms = [Math.max(0, rotated1[0]), Math.max(0, rotated2[0])] Uses: rotated1, rotated2

Returns

Example result

directTerms: array (2 items): [1, 2]
rotatedTerms: array (2 items): [1, 2]
directSum: 3
rotatedSum: 3

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

The source absorbs scale and remaining parameters into each function ρⱼ and takes wⱼ to be a unit vector. A rotation maps each direction to the first basis vector. The tensor factor is constant in the remaining coordinates.

Source: Source expression