The Modern Mathematics of Deep Learning

A depth-width condition

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Approximation and depth · Study approximation error and network size under stated conditions.

The project supplies finite observations and fitted parameters. A held-out sample does not establish population bounds or prove a theorem.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 3.2 Simple pieces, many layers

The theorem gives a lower bound on the best uniform approximation error when a ReLU network is too small under a stated width condition.

The code calculates that width threshold for illustrative inputs.

For certain smooth non-affine functions, restricted depth and size limit approximation accuracy.

Source formula example

\inf_{\theta\in{\mathbb{R}}^{P(N)}}\|\Phi_{{a}}(\cdot,\theta)-g\|_{L^{\infty}([0,1]^{d})}\geq{\varepsilon}.

illustration · The code evaluates the stated size condition with an illustrative constant. It does not construct a network or compute its best approximation error.

Symbols

$g$: A twice continuously differentiable non-affine function on [0,1]ᵈ.
$L$: Integer network depth at least 2.
$ε$: Positive error level below 1.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Illustrative constant cAccuracy εDepth LNeuron count

This example’s data flow

You choose

input.c = 1
input.epsilon = 0.25
input.depth = 2
input.neurons = 2

Calculated here

Result expression: { threshold, sizeCondition: neurons <= threshold, statedErrorLowerBound: epsilon, }

c = input.c Uses: input.c
epsilon = input.epsilon Uses: input.epsilon
depth = input.depth Uses: input.depth
neurons = input.neurons Uses: input.neurons
threshold = c * epsilon ** (-1 / (2 * (depth - 1))) Uses: c, epsilon, depth

Returns

Example result

threshold: 2
sizeCondition: true
statedErrorLowerBound: 0.25

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

For each positive dimension d, depth L ≥ 2 and non-affine g in C²([0,1]ᵈ), the source asserts that a positive c exists. For every ε in (0,1) and every scalar-output ReLU architecture with L layers and total neuron count ‖N‖₁ at most c ε^(−1/(2(L−1))), the best uniform approximation error is at least ε. The value of c is not supplied by the source and the example does not establish the hypotheses.

Source: Source expression