The Modern Mathematics of Deep Learning

Exponential loss and margin

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Optimization theory · Study convergence and loss geometry under assumptions.

The project supplies finite observations and fitted parameters. A held-out sample does not establish population bounds or prove a theorem.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 2.3 The path matters

The source studies exponential loss and a rescaled risk. As β grows, the rescaled risk approaches the smallest margin.

The toy uses margins 1 and 2 before scaling and their mean exponential loss.

This limit connects a loss objective with the smallest training margin.

Source formula example

r_{\beta}(\theta)=-\frac{\log(r(\beta\theta))}{\beta}\to\gamma(\theta),\quad\beta\to\infty.

illustration · The code evaluates one finite example. It does not prove the limiting statement, and very large β can cause exponential underflow.

Symbols

$β$: Positive scale tending to infinity in the limit.
$γ(θ)$: Minimum training margin for θ.
$r$: Empirical risk using exponential margin loss in this source discussion.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Scale β

This example’s data flow

You choose

input.beta = 1

Initial values in code

margins = [1, 2]

Calculated here

Result expression: { risk, rescaledRisk: -Math.log(risk) / beta, minimumMargin: Math.min(...margins), }

beta = input.beta Uses: input.beta
risk = margins.reduce((sum, margin) => sum + Math.exp(-beta * margin), 0) / margins.length Uses: margins, beta

Returns

Example result

risk: ≈0.2516
rescaledRisk: ≈1.38
minimumMargin: 1

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

The source specializes to the exponential margin loss ℓ(t)=exp(−t). The risk is its mean over a finite sample, and the limit β→∞ yields the minimum of the finitely many margins. This code does not claim the result for arbitrary loss functions.

Source: Source expression