The Modern Mathematics of Deep Learning

Exponential risk and its gradient

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Optimization theory · Study convergence and loss geometry under assumptions.

The project supplies finite observations and fitted parameters. A held-out sample does not establish population bounds or prove a theorem.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 2.3 The path matters

For exponential margin loss ℓ(z)=exp(−z), its derivative is −exp(−z). The source uses this to analyze gradient flow on separable data.

The toy has one sample x=1, y=1 and a separating parameter θ-hat=1.

The derivative connects loss reduction to margin growth.

Source formula example

\big\langle\hat{\theta},\nabla_{\theta}r(\theta)\rangle=\frac{1}{m}\sum_{i=1}^{m}\underbrace{\ell^{\prime}(y^{(i)}\langle\theta,x^{(i)}\rangle)}_{<0}\underbrace{y^{(i)}\langle\hat{\theta},x^{(i)}\rangle}_{>0}.

illustration · One scalar sample gives an exact toy calculation, not a proof of the general limiting or gradient-flow statements.

Symbols

$r$: Mean exponential margin loss over the sample.
$β$: Positive scale.
$θ̂$: A separating parameter with positive margins.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

θβ

This example’s data flow

You choose

input.theta = 1
input.beta = 2

Calculated here

Result expression: { risk, gradient, derivative, gradientDotSeparator: gradient, scaledRisk, smoothedMargin, flowRate, thetaNorm: Math.abs(theta), }

theta = input.theta Uses: input.theta
beta = input.beta Uses: input.beta
risk = Math.exp(-theta) Uses: theta
gradient = -risk Uses: risk
derivative = -Math.exp(-theta) Uses: theta
scaledRisk = Math.exp(-beta * theta) Uses: beta, theta
smoothedMargin = -Math.log(scaledRisk) / beta Uses: scaledRisk, beta
flowRate = -gradient / risk Uses: gradient, risk

Returns

Example result

risk: ≈0.3679
gradient: ≈-0.3679
derivative: ≈-0.3679
gradientDotSeparator: ≈-0.3679
scaledRisk: ≈0.1353
smoothedMargin: 1
flowRate: 1
thetaNorm: 1

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

For a linearly separable sample the source takes ℓ(z)=exp(−z), defines rβ(θ)=−log(r(βθ))/β, and evolves θ by the gradient of r1, equal to −∇r/r. The toy sample has one positive unit input and θ-hat=1, so its margin is θ and the smoothed margin is exactly θ. Along θ→+∞ the norm grows without bound and the loss gradient approaches zero; the finite control only evaluates selected values.

Source: Source expression