The Modern Mathematics of Deep Learning

A Sobolev approximation bound

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Approximation and depth · Study approximation error and network size under stated conditions.

The project supplies finite observations and fitted parameters. A held-out sample does not establish population bounds or prove a theorem.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 1.2.2 Can the network express the answer?

A best approximation error is bounded by a constant, a width factor, and the function’s smoothness norm.

This example makes the source statement computable on a finite model.

Source formula example

\inf_{\theta^{(2)}\in{\mathbb{R}}^{1\times n}\times{\mathbb{R}}}\|\Phi_{{a}}(\cdot,(\theta^{(1)},\theta^{(2)}))-g\|_{L^{p}((0,1)^{d})}\leq cn^{-\frac{d}{k}}\|g\|_{W^{k,p}((0,1)^{d})}.

illustration · A zero-function case cannot verify the general approximation rate or identify c.

Symbols

$g$: Function in Wᵏ,ᵖ((0,1)ᵈ).
$n$: Positive hidden width.
$d,k$: Positive dimension and smoothness order.
$p$: Exponent in [1,∞].
$c$: Positive existence constant, not numerically specified.
$θ⁽¹⁾$: Hidden parameters selected in the theorem.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Hidden units

This example’s data flow

You choose

input.width = 2

Initial values in code

functionNorm = 0
bestError = 0

Calculated here

Result expression: { bestError, rateFactor, functionNorm, upperBound: 0, theoremConstant: null }

n = input.width Uses: input.width
rateFactor = n ** -1 Uses: n

Returns

Example result

bestError: 0
rateFactor: 0.5
functionNorm: 0
upperBound: 0
theoremConstant: null

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

Set d=k=1, take g=0 and zero output parameters. The nonnegative best error and Sobolev norm are both zero; the bound is zero regardless of the unknown c. the source theorem assumes a smooth nonpolynomial activation and states that for every n some first-layer parameters work for every g in the specified Sobolev space. The exponent is transcribed as printed in the source.

Source: Source expression