The Modern Mathematics of Deep Learning

Counting network parameters

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Model classes and structure · Describe families of rules, their size and structural properties.

The project supplies finite observations and fitted parameters. A held-out sample does not establish population bounds or prove a theorem.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 1.2 Learning from examples · 1.2.3 Does it work on new examples? · 1.3 The questions ahead · More source sections

Each layer has one weight per input-output pair and one bias per output. Add those counts over all layers.

The parameter count describes the size of a fully connected network.

Source formula example

P(N)\coloneqq\sum_{\ell=1}^{L}N_{\ell}N_{\ell-1}+N_{\ell}

exact · The result is exact for these finite toy inputs, apart from ordinary computer rounding.

Symbols

$N_ℓ$: Positive integer width of layer ℓ.
$L$: Number of layers with parameters.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Hidden units

This example’s data flow

You choose

input.hidden = 3

Calculated here

Result expression: { counts, total: counts.reduce((a, b) => a + b, 0) }

widths = [2, input.hidden, 1] Uses: input.hidden
counts = widths.slice(1).map((width, i) => width * widths[i] + width) Uses: widths

Returns

Example result

counts: array (2 items): [9, 4]
total: 13

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

For widths [2, 3, 1], the first layer has 2×3 weights and 3 biases; the second has 3 weights and 1 bias.

Source: Source expression