The Modern Mathematics of Deep Learning

Size for smooth-function approximation

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Approximation and depth · Study approximation error and network size under stated conditions.

The project supplies finite observations and fitted parameters. A held-out sample does not establish population bounds or prove a theorem.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 3.2 Simple pieces, many layers

The theorem bounds the number of network parameters needed for a stated approximation accuracy. The code calculates the right side from supplied inputs.

The formula shows how the permitted parameter count depends on dimension and accuracy.

Source formula example

P(N)\leq c{\varepsilon}^{-d/(k-s)}\log(1/{\varepsilon})

illustration · The constant is chosen for arithmetic; the theorem does not provide its value here. The code evaluates a bound and constructs no network.

Symbols

$P(N)$: Number of parameters in the architecture.
$d$: Positive input dimension.
$k,s$: Function regularity and error-norm orders.
$ε$: Target error in (0,1/2).

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Constant cAccuracy εDimension dRegularity kError order s

This example’s data flow

You choose

input.c = 1
input.epsilon = 0.25
input.d = 2
input.k = 2
input.s = 0

Calculated here

Result expression: { upperBound: c * epsilon ** (-d / (k - s)) * Math.log(1 / epsilon) }

c = input.c Uses: input.c
epsilon = input.epsilon Uses: input.epsilon
d = input.d Uses: input.d
k = input.k Uses: input.k
s = input.s Uses: input.s

Returns

Example result

upperBound: ≈5.545

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

The theorem assumes positive integers d and k with k ≥ 2, p in [1,∞], s in [0,1], B > 0, and a piecewise-linear activation with at least one breakpoint. For every ε in (0,1/2), it asserts existence of an architecture obeying this size bound; for every g in W^{k,p}((0,1)^d) with Sobolev norm at most B, some parameters achieve W^{s,p} error at most ε. The positive constant c is asserted to exist and depends on the theorem setting.

Source: Source expression