The Modern Mathematics of Deep Learning

Averaging optimization iterates

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Learning and parameter updates · Select a rule or update parameters using a learning rule.

The current objective supplies a gradient; training updates the model parameters.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 1.2 Learning from examples · 1.2.1 Finding better settings

The source optimization theorem averages a sequence of parameter vectors and compares the result with a minimum in the unit ball.

The toy uses r(θ)=θ², whose minimum in (−1,1) is θ*=0, and three explicitly computed gradient steps.

Averaging iterates is one way to form an output from an optimization sequence.

Source formula example

\bar\Theta=\frac1K\sum_{k=1}^K\Theta^{(k)},\quad\theta^*\in\operatorname{argmin}_{\theta\in B_1(0)}r(\theta)

illustration · This evaluates a deterministic one-dimensional quadratic. It does not establish the source stochastic optimization bound.

Symbols

$K$: Positive number of iterates.
$Θ^{(k)}$: Parameter at step k.
$η_k$: Step size at iteration k.
$r$: Real-valued objective on the parameter domain.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Starting θ

This example’s data flow

You choose

input.start = 0.8

Initial values in code

rate = 0.25
iterates = []

Calculated here

Result expression: { rate, iterates, average, averageRisk: average ** 2, minimizer: 0, minimumRisk: 0, }

start = input.start Uses: input.start
theta = start Uses: start
for (let k = 0; k < 3; k++) { theta = theta - rate * 2 * theta; iterates.push(theta); } Uses: theta, rate, iterates
average = iterates.reduce((sum, value) => sum + value, 0) / iterates.length Uses: iterates

Returns

Example result

rate: 0.25
iterates: array (3 items): [0.4, 0.2, 0.1]
average: ≈0.2333
averageRisk: ≈0.05444
minimizer: 0
minimumRisk: 0

Some runtime paths or values are not resolved here.

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

The source theorem uses a real objective on a unit ball and an averaged iterate. This toy’s objective is defined on all real numbers, its minimizer is exactly zero, and every displayed iterate stays inside the one-dimensional unit ball.

Source: Averaged iterates