The Modern Mathematics of Deep Learning

Feature approximation error

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Approximation and depth · Study approximation error and network size under stated conditions.

The project supplies finite observations and fitted parameters. A held-out sample does not establish population bounds or prove a theorem.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 5.1 The shape of training loss

A small feature change causes at most the output-weight magnitude times that change in a one-unit network.

This calculation explains the corresponding source expression.

Source formula example

{\mathcal{O}}({\varepsilon})

illustration · This explicitly bounded one-unit toy does not establish the general feature-covering or risk-path argument.

Symbols

$ε$: Nonnegative approximation tolerance.
$O(ε)$: A quantity bounded by a constant times ε in the stated local regime.
$θ⁽²⁾$: Network output weights.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Feature error ε

This example’s data flow

You choose

input.epsilon = 0.1

Initial values in code

weight = 2
oldFeature = 1

Calculated here

Result expression: { epsilon, difference, bound: Math.abs(weight) * epsilon, oldRisk, newRisk, riskDifference, riskBound: 12 * epsilon, }

epsilon = input.epsilon Uses: input.epsilon
newFeature = 1 + epsilon Uses: epsilon
difference = Math.abs(weight * newFeature - weight * oldFeature) Uses: weight, newFeature, oldFeature
oldRisk = (weight * oldFeature) ** 2 Uses: weight, oldFeature
newRisk = (weight * newFeature) ** 2 Uses: weight, newFeature
riskDifference = newRisk - oldRisk Uses: newRisk, oldRisk

Returns

Example result

epsilon: 0.1
difference: 0.2
bound: 0.2
oldRisk: 4
newRisk: 4.84
riskDifference: 0.84
riskBound: 1.2

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

The invented feature values differ by ε, the output weight is 2, and the target is zero. The output error is 2ε. The squared-risk increase is 8ε+4ε² ≤ 12ε for 0 ≤ ε ≤ 1. The source uses local Lipschitz continuity of risk and bounded output-weight L1 norm after feature replacement; its implicit constant is not specified.

Source: Source expression