The Modern Mathematics of Deep Learning

Gradient flow and the kernel

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Kernels and linearization · Relate parameter gradients or features to kernel matrices.

The project supplies finite observations and fitted parameters. A held-out sample does not establish population bounds or prove a theorem.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 2.1 A view through similarity

Gradient flow changes parameters in the negative risk-gradient direction. Each Dᵢ is the derivative of the loss with respect to the prediction.

For the toy model Φ(x,θ) = θx and squared loss, the code computes the parameter, prediction, and risk rates at one instant.

The kernel expresses how a parameter update changes predictions at different inputs.

Source formula example

\frac{\mathrm{d}\Theta(t)}{\mathrm{d}t}=-\nabla_{\theta}\widehat{\mathcal{R}}_{s}(\Phi(\cdot,\Theta(t)))=-\frac{1}{m}\sum_{i=1}^{m}\nabla_{\theta}\Phi(x^{(i)},\Theta(t))D_{i}(t),\quad\Theta(0)=\Theta_{0},

illustration · This computes instantaneous rates for one smooth scalar model and two fixed samples. It does not solve the full trajectory or show that a general kernel stays constant.

Symbols

$D_i$: Derivative of the loss with respect to the prediction on training example i.
$K_θ(x,x^{\prime})$: Dot product of output gradients at two inputs.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Parameter θQuery x

This example’s data flow

You choose

input.theta = 1
input.query = 1

Initial values in code

xs = [1, 2]
ys = [2, 4]

Calculated here

Result expression: { predictions, lossDerivatives, parameterRate, kernelRow, kernelMatrix, predictionRate, riskRate, }

predictions = xs.map((x) => input.theta * x) Uses: xs, input.theta
lossDerivatives = predictions.map((value, i) => 2 * (value - ys[i])) Uses: predictions, ys
parameterRate = -xs.reduce((sum, x, i) => sum + x * lossDerivatives[i], 0) / xs.length Uses: xs, lossDerivatives
kernelRow = xs.map((x) => input.query * x) Uses: xs, input.query
kernelMatrix = xs.map((xi) => xs.map((xj) => xi * xj)) Uses: xs
predictionRate = -kernelRow.reduce((sum, value, i) => sum + value * lossDerivatives[i], 0) / xs.length Uses: kernelRow, lossDerivatives, xs
riskRate = -xs.reduce( (sum, xi, i) => sum + xs.reduce( (inner, xj, j) => inner + lossDerivatives[i] * xi * xj * lossDerivatives[j], 0, ), 0, ) / xs.length ** 2 Uses: xs, lossDerivatives

Returns

Example result

predictions: array (2 items): [1, 2]
lossDerivatives: array (2 items): [-2, -4]
parameterRate: 5
kernelRow: array (2 items): [1, 2]
kernelMatrix: array (2 items): [array (2 items): [1, 2], array (2 items): [2, 4]]
predictionRate: 5
riskRate: -25

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

The source uses the chain rule to express parameter velocity, output velocity and risk velocity through D and the neural tangent kernel. This linear model has gradient ∂Φ/∂θ = x, hence kernel K(x,x′) = xx′. For squared loss, Dᵢ = 2(Φ(xᵢ,θ)−yᵢ). The risk rate equals minus the squared parameter gradient in this scalar example.

Source: Source expression