The Modern Mathematics of Deep Learning

Surrogate scores and classifiers

Penguin learning project

The project uses 342 complete Palmer penguin records: 240 training-pool and 102 held-out rows. The selected training rows determine feature scaling and model parameters. Source CSV

This project trains weights with softmax, unlike the homepage lab, which uses nearest species averages. Same penguin measurements; different classification methods. Predictions can differ.

Prediction rules and layers · Describe or apply prediction rules, layers and parameters.

The project uses a linear rule with two features. The fitted weights produce scores or normalized mass.

Consider this concept alongside the project; its source assumptions may differ. These observations do not prove a theorem.

Input: 12 × 2; Held-out evaluation: 102 · #58 · 40.6 mm, 193 mm. Known answer: Adélie; Prediction: Adélie · [≈0.7383, ≈0.2444, ≈0.01737].

Used in these source sections: 1.2 Learning from examples

The source describes choosing a real-valued rule by a surrogate loss, then applying the sign function to obtain a classifier.

The toy has one positive example and two candidate scores. It selects the lower logistic loss and compares the resulting zero-one risk.

A differentiable score loss can be used while fitting a classification model.

Source formula example

{\operatorname{sgn}}(\widehat{f}_{s}^{\mathrm{surr}})\in\argmin_{f\in{\mathcal{F}}_{{{a}},{\operatorname{sgn}}}}\widehat{\mathcal{R}}_{s}(f)

illustration · The code checks one finite case. Minimizing a surrogate loss over an arbitrary restricted class does not automatically guarantee an exact empirical zero-one minimizer.

Symbols

$ℱ_a$: A class of real-valued network scores.
$ℱ_{a,sgn}$: Class obtained by applying sign to those scores.
$ℒ^{surr}$: A surrogate loss on real-valued scores and examples.

Source formula example

This source example uses its own inputs and assumptions. Editing it does not change the project.

Edit a value or the code, then run the example. This is a finite demonstration, not a proof of the paper’s theorem.

Second score

This example’s data flow

You choose

input.score = 1

Initial values in code

label = 1

Calculated here

Result expression: { scores, surrogateLosses, predictions, risks, selected, selectedRisk: risks[selected], minimumRisk: Math.min(...risks), }

scores = [-1, input.score] Uses: input.score
surrogateLosses = scores.map((score) => Math.log(1 + Math.exp(-label * score))) Uses: scores, label
predictions = scores.map((score) => (score >= 0 ? 1 : -1)) Uses: scores
risks = predictions.map((prediction) => (prediction === label ? 0 : 1)) Uses: predictions, label
selected = surrogateLosses.indexOf(Math.min(...surrogateLosses)) Uses: surrogateLosses

Returns

Example result

scores: array (2 items): [-1, 1]
surrogateLosses: array (2 items): [≈1.313, ≈0.3133]
predictions: array (2 items): [-1, 1]
risks: array (2 items): [1, 0]
selected: 1
selectedRisk: 0
minimumRisk: 0

JavaScript

Read numbers from input and return the result. JavaScript basics.

Result

Enable JavaScript to run this example.

Technical statement

The source presents the minimizer relation as a property of the surrogate setting and mentions logistic loss. This toy satisfies the relation. The displayed result is not a proof that all surrogate choices and restricted classes have that property.

Source: Source expression