The Modern Mathematics of Deep Learning

Feature change after parameter movement

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Optimization theory · Study convergence and loss geometry under assumptions.

The project supplies finite observations and fitted parameters. A held-out sample does not establish population bounds or prove a theorem.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 5.2 When parameters move only a little

The same output-weight gradient formula can be evaluated after a parameter change, using the new features and errors.

The comparison checks whether a lower gradient bound survives a parameter change.

Source formula example

\begin{split}\|\nabla_{\theta}{r}(\Theta+\bar{\theta})\|_{2}^{2}&\geq\frac{4}{m^{2}}\Big\|\sum_{i=1}^{m}\nabla_{\theta^{(2)}}\Phi(x^{(i)},\Theta+\bar{\theta})(\Phi(x^{(i)},\Theta+\bar{\theta})-y^{(i)})\Big\|_{2}^{2}\\ &\!\!\underset{\text{(5.5)}}{=}\frac{4}{m^{2}}\Big\|\sum_{i=1}^{m}(\nabla_{\theta^{(2)}}\Phi(x^{(i)},\Theta)+{\mathcal{O}}(1))(\Phi(x^{(i)},\Theta+\bar{\theta})-y^{(i)})\Big\|_{2}^{2}\\ &\underset{(*)}{\gtrsim}\frac{1}{m^{2}}(\lambda_{\min}(\bar{K}_{\Theta})+{\mathcal{O}}(1))\|(\Phi(x^{(i)},\Theta+\bar{\theta})-y^{(i)})_{i=1}^{m}\|_{2}^{2}\\ &\gtrsim\frac{n}{m}{r}(\Theta+\bar{\theta}),\end{split}

illustration · This special perturbation does not establish stability for general parameter changes or the source convergence theorem.

Symbols

$Θ$: Initialization parameters.
$θ̄$: Parameter displacement.
$r$: Squared training loss.
$K̄$: Output-gradient Gram matrix at initialization.
$O(1), ≳$: Asymptotic terms with constants subject to the source assumptions.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Output-weight displacement

This example’s data flow

You choose

input.change = 0.5

Initial values in code

initialization = [1, 1]

Calculated here

Result expression: { initialization, displacement, perturbed, risk, fullSquaredNorm: fullGradient.reduce((s, v) => s + v * v, 0), outputSquaredNorm: outputGradient ** 2, initialGram: 1, perturbedGram: 1, eigenLowerBound: 4 * risk, width...

a = 1 + input.change Uses: input.change
error = a Uses: a
risk = error ** 2 Uses: error
outputGradient = 2 * error Uses: error
fullGradient = [outputGradient, 2 * error * a] Uses: outputGradient, error, a
displacement = [input.change, 0] Uses: input.change
perturbed = [a, 1] Uses: a

Returns

Example result

risk: 2.25
fullSquaredNorm: 29.25
outputSquaredNorm: 9
initialGram: 1
perturbedGram: 1
eigenLowerBound: 9
widthScaledRisk: 2.25
chosenConstant: 4

… (11 items)

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

For Φ = awx with one x = 1, target 0 and w = 1, change a from 1 to 1 + displacement. The feature and its Gram eigenvalue stay constant, so the feature-change term is zero. The final comparison has the chosen toy constant 4; the source uses asymptotic O(1) terms and a high-probability lower bound under its own assumptions. The source’s training discussion assumes a smooth, non-affine activation, distinct training inputs, quadratic loss, and independent Gaussian hidden and output initial parameters with variance 1/n. Its high-probability and convergence statements require the additional width, neighborhood, and step-size conditions stated in that discussion. The source marks the Gram-eigenvalue comparison step by (*); this toy evaluates that comparison explicitly.

Source: Source expression