The Modern Mathematics of Deep Learning

A depth separation statement

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Approximation and depth · Study approximation error and network size under stated conditions.

The project supplies finite observations and fitted parameters. A held-out sample does not establish population bounds or prove a theorem.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 3.1 A rule based on distance

The theorem constructs a three-layer target for which all sufficiently narrow two-layer networks have error at least c.

The code checks the numerical width condition for chosen illustrative constants.

A deeper network can represent some functions much more efficiently than a shallower one.

Source formula example

\inf_{\theta\in{\mathbb{R}}^{P((d,n,1))}}\|\Phi_{((d,n,1),\varrho)}(\cdot,\theta)-\Phi_{{{a}}}(\cdot,\theta^{*})\|_{L^{2}(\mu)}\geq c.

illustration · The constants are illustrative, including C = 1 in the dimension check. Checking one width condition does not construct the target, measure its error, or prove the theorem.

Symbols

$μ$: A probability measure constructed by the theorem.
$c,C$: Positive constants whose existence is asserted.
$n$: Number of hidden units in the two-layer approximating network.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Illustrative constant cDimension dWidth n

This example’s data flow

You choose

input.c = 1
input.d = 2
input.n = 4

Calculated here

Result expression: { widthLimit, widthCondition: input.n <= widthLimit, dimensionCondition: input.d >= 1, illustrativeC: 1, statedLowerBound: input.c, }

widthLimit = input.c * Math.exp(input.c * input.d) Uses: input.c, input.d

Returns

Example result

widthLimit: ≈7.389
widthCondition: true
dimensionCondition: true
illustrativeC: 1
statedLowerBound: 1

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

The source allows ReLU, logistic or Heaviside activation and asserts positive constants c,C. For every positive integer d ≥ C, it constructs a probability measure on ℝᵈ and a three-layer target with widths bounded by Cd⁵, parameter magnitudes bounded by Cd^C, and output magnitude bounded by 2. For every n ≤ c exp(cd), the best two-layer approximation has L²(μ) error at least c. The theorem does not assert this for every target or every probability measure.

Source: Source expression