The Modern Mathematics of Deep Learning

Covering classifiers and their losses

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Generalization and risk bounds · Relate sample behavior to population risk under stated assumptions.

The project supplies finite observations and fitted parameters. A held-out sample does not establish population bounds or prove a theorem.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 1.2 Learning from examples

For binary labels and predictions ±1, zero-one loss is (1−yf(x))/2. The distance between two losses is half the distance between their predictions on the sample.

This toy has one labeled input and two constant classifiers +1 and −1.

This lets the source translate a loss-class covering problem into a classifier covering problem.

Source formula example

N_{\alpha}({\mathcal{G}},d_{S})=N_{2\alpha}({\mathcal{F}},d_{(X^{(i)})_{i=1}^{m}}).

illustration · The covering counts are exact for this two-function finite class. The separate VC expression omits the hidden constant in ≲ and is not an exact upper bound by itself.

Symbols

$ℱ$: Binary classifier class with values in {−1,1}.
$\mathcal{G}$: Associated zero-one loss class.
$α$: Positive covering radius; the VC estimate uses α in (0,1).

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Radius α

This example’s data flow

You choose

input.alpha = 0.5

Initial values in code

predictions = [-1, 1]
label = 1
vcDimension = 1

Calculated here

Result expression: { predictions, losses, lossDistance, predictionDistance, lossCover, classifierCover, logCoverAtAlpha: Math.log(classifierCoverAtAlpha), hausslerScale: alpha < 1 ? vcDimension * Math.log(1 / alpha) : null, }

alpha = input.alpha Uses: input.alpha
losses = predictions.map((prediction) => (1 - label * prediction) / 2) Uses: predictions, label
predictionDistance = Math.abs(predictions[0] - predictions[1]) Uses: predictions
lossDistance = Math.abs(losses[0] - losses[1]) Uses: losses
lossCover = lossDistance < alpha ? 1 : 2 Uses: lossDistance, alpha
classifierCover = predictionDistance < 2 * alpha ? 1 : 2 Uses: predictionDistance, alpha
classifierCoverAtAlpha = predictionDistance < alpha ? 1 : 2 Uses: predictionDistance, alpha

Returns

Example result

predictions: array (2 items): [-1, 1]
losses: array (2 items): [1, 0]
lossDistance: 1
predictionDistance: 2
lossCover: 2
classifierCover: 2
logCoverAtAlpha: ≈0.6931
hausslerScale: ≈0.6931

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

The source identity assumes binary predictions and labels in {−1,1} and the indicated sample pseudometrics. The VC estimate bounds log covering number up to a hidden constant for α∈(0,1). The two constant classifiers shatter one point but cannot assign opposite labels to two points, so their VC dimension is 1.

Source: Source expression