The Modern Mathematics of Deep Learning

Hidden features and interpolation

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Features · Transform inputs into features used by a rule.

The project scales bill and flipper measurements using training-row means and scales; its feature vector has two entries.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 1.2.2 Can the network express the answer? · 1.2 Learning from examples

Put the hidden-unit outputs for each training input in one column. In the source proof, an invertible square submatrix A lets output weights be chosen to reproduce the target labels.

The toy uses inputs 0 and 1 and features ReLU(x), ReLU(1−x). Its feature matrix is its own inverse.

A suitable feature matrix lets a network fit a finite set of distinct examples exactly.

Source formula example

\widetilde{A}\coloneqq\begin{bmatrix}\varrho(\widetilde{W}^{(1)}(x^{(1)})+\tilde{b}^{(1)}))\dots\varrho(\widetilde{W}^{(1)}(x^{(m)})+\tilde{b}^{(1)}))\end{bmatrix}\in{\mathbb{R}}^{n\times m}.

illustration · This constructs one small example with a known inverse. It does not establish invertibility of arbitrary feature matrices or prove the general interpolation proposition.

Symbols

$\tilde{A}$: Matrix with n hidden units as rows and m training points as columns.
$A$: An invertible m-by-m submatrix selected in the proof.
$W^{(2)}$: Output weight matrix, with one column per hidden feature.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

First target

This example’s data flow

You choose

input.firstLabel = 2

Initial values in code

inputs = [0, 1]
inverse = [ [0, 1], [1, 0], ]

Calculated here

Result expression: { matrix, inverse, labels, outputWeights, outputBias: 0, proofOutputMatrix, matrixProduct, predictions, }

labels = [input.firstLabel, 3] Uses: input.firstLabel
features = inputs.map((x) => [Math.max(0, x), Math.max(0, 1 - x)]) Uses: inputs
matrix = [ [features[0][0], features[1][0]], [features[0][1], features[1][1]], ] Uses: features
outputWeights = inverse[0].map((_, j) => labels.reduce((sum, y, i) => sum + y * inverse[i][j], 0), ) Uses: inverse, labels
predictions = features.map((column) => column.reduce((sum, value, i) => sum + value * outputWeights[i], 0), ) Uses: features, outputWeights
proofOutputMatrix = inverse Uses: inverse
matrixProduct = proofOutputMatrix.map((row) => matrix[0].map((_, j) => row.reduce((sum, value, k) => sum + value * matrix[k][j], 0)), ) Uses: proofOutputMatrix, matrix

Returns

Example result

matrix: array (2 items): [array (2 items): [0, 1], array (2 items): [1, 0]]
inverse: array (2 items): [array (2 items): [0, 1], array (2 items): [1, 0]]
labels: array (2 items): [2, 3]
outputWeights: array (2 items): [3, 2]
outputBias: 0
proofOutputMatrix: array (2 items): [array (2 items): [0, 1], array (2 items): [1, 0]]
matrixProduct: array (2 items): [array (2 items): [1, 0], array (2 items): [0, 1]]
predictions: array (2 items): [2, 3]

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

The source proposition requires distinct inputs and a continuous non-polynomial activation. It asserts that suitable first-layer parameters exist for any labels; the proof then chooses W² = [y¹ … yᵐ]A⁻¹. The toy computes a particular ReLU instance and exact finite matrix inverse. Its proof network has architecture (1,2,2), first-layer weights [1,−1], biases [0,1], a two-by-two output matrix and zero output biases. Multiplying its output matrix by the feature matrix gives the two-by-two identity. The separate final label fit uses one output and zero output bias.

Source: Source expression