The Modern Mathematics of Deep Learning

Training loss and a robust margin

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Risk and assessment · Distinguish loss on a sample from expected loss.

The fitted rule is evaluated on 102 held-out observations, not used for fitting.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 2.3 The path matters

The source defines empirical classification risk r and the smallest training margin γ. It relates a zero-loss neighborhood of a solution to a margin bound.

For one example x=1, y=1 and positive θ-hat, all perturbations of size less than δ<θ-hat keep the score positive.

Robustness to parameter changes is connected to training margins.

Source formula example

r(\theta)\coloneqq\widehat{\mathcal{R}}_{s}(\langle\theta,\cdot\rangle)\quad\text{and}\quad\gamma(\theta)\coloneqq\min_{i\in[m]}M(\langle\theta,\cdot\rangle,z^{(i)})=\min_{i\in[m]}y^{(i)}\langle\theta,x^{(i)}\rangle

illustration · The neighborhood maximum and boundary perturbation are calculated analytically for this one-example model under the stricter condition 0 < δ < θ-hat. This does not prove the general source inequality.

Symbols

$r$: Empirical zero-one risk of the linear classifier.
$γ$: Smallest training margin.
$δ$: Positive perturbation radius.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

θ-hatRadius δ

This example’s data flow

You choose

input.theta = 2
input.delta = 1

Calculated here

Result expression: { margin, zeroOneLoss, maximumNeighborhoodLoss: 0, worstPerturbedMargin, upperExpression: margin - delta, }

theta = input.theta Uses: input.theta
delta = input.delta Uses: input.delta
margin = theta Uses: theta
worstPerturbedMargin = theta - delta Uses: theta, delta
zeroOneLoss = theta >= 0 ? 0 : 1 Uses: theta

Returns

Example result

margin: 2
zeroOneLoss: 0
maximumNeighborhoodLoss: 0
worstPerturbedMargin: 1
upperExpression: 1

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

The source assumes binary classification with zero-one loss and linearly separable data, then relates a zero-risk neighborhood of θ-hat to a margin. The toy uses one nonzero input and a positive margin, with δ strictly below that margin.

Source: Source expression