Beyond the practice set

Changing one weight can move predictions for several inputs at once. For example, if the first input number is 1 in one case and 2 in another, raising its weight by 1 raises those predictions by 1 and 2. Features tell us how strongly the predictions move together.

To compare two lists, multiply matching numbers and add: (1, 2) and (2, 1) give 1×2 + 2×1 = 2 + 2 = 4. The paper’s neural tangent kernel compares lists of prediction changes this way. The ribbon activity instead scores how close a length is to 5. Neither score is a probability or a distance.

For the rule Φ(x, θ) = θ₁x₁ + θ₂x₂, θ is the list of weights. The list (x₁, x₂) says how much the prediction changes when each weight rises by one. This list of rates is called the gradient.

function similarity(firstGradient, secondGradient) {
  const similarity = firstGradient[0] * secondGradient[0]
    + firstGradient[1] * secondGradient[1];
  return { similarity };
}
const result = similarity([1, 2], [2, 1]);

The neural tangent kernel compares these gradients for two inputs. For very wide networks under certain initialization and training conditions, that comparison changes little as training continues.

In that regime, the lists describing how predictions change with the weights stay nearly fixed. The learner then behaves approximately like a simpler model using those nearly fixed features, even though the network’s output can be nonlinear in its inputs. This is called the kernel regime or lazy training. It describes a particular setting, not every useful network.

Counting weights is a crude measure of how freely a network behaves. A large network with small or otherwise controlled weights can differ from one whose weights become enormous.

A score of 6 predicts +1; a score of −6 predicts −1. Multiply the score by the correct label y, which is +1 or −1. The result, M = y×f(x), is called the raw margin. A positive margin means the score chose the correct side; a bigger positive number gives a bigger numerical cushion. Zero is the special boundary.

For f(x) = wx with no bias, dividing the raw margin by |w| gives a distance with a sign: positive on the correct side of the boundary. With weight 2 and a positive example at 3, the raw margin is 6. Divide 6 by 2 to get 3 units from the boundary at 0.

function classificationMargin(weight, value) {
  if (weight === 0) throw new Error('Weight must be nonzero');
  const score = weight * value;
  const norm = Math.abs(weight);
  return { score, norm, margin: score / norm };
}
const result = classificationMargin(2, 3);

A toy cat score of 7 points versus a dog score of 5 leaves a gap of 2 points. If you move the cat score to 3, the gap becomes −2: the dog wins. The illustration shows a score gap; the paper’s signed margin uses the label multiplied by a raw score.

If we double a positive weight, the score and raw margin double, but the boundary stays at 0 and the distance stays 3. The paper studies bounds based on weight sizes and margins under stated assumptions; a large margin alone does not guarantee success on new examples.

Two networks can make equally few training mistakes, yet predict different answers elsewhere. The process used to find weights can influence which of the many fitting answers we obtain.

Compare two toy rules that both fit the training point (0, 0) but disagree at a new input. This shows a choice among fitting rules. The paper’s result concerns a different task: classification where a straight boundary can separate the two labels.

Two toy rules both return 0 at the only training input, 0. At a new input, one returns 0 and the other returns the input.

function compareRules(newInput) {
  return { trainingA: 0, trainingB: 0, newA: 0, newB: newInput };
}
const result = compareRules(1);

An explicit penalty is a cost we add to the training score. An implicit preference comes from the way the search moves, even when we add no such cost. For example, a search may favor one of several rules that fit. Under special conditions, gradient descent on simple linear models tends toward small weights or a large margin.

The paper also discusses the shape of nearby loss values, or flatness. These ideas offer possible explanations for generalization but do not establish that any particular deep network will find the best generalizing solution.

With too few adjustable numbers a model may miss a real pattern. Giving it more freedom often helps at first. Classical teaching suggests that still more freedom eventually causes trouble by fitting noise.

But some models show a different curve. Error on new examples can rise near the point where the model first fits all practice labels, then fall again as the model grows larger. The paper calls this double descent.

For the two values −1 and +1, the average is 0. Their squared distances from 0 are 1 and 1, with average 1. That average is called variance. The paper studies independently drawn random features with variance 1. Their true weights are zero, so they carry no target signal. Random noise also has variance 1. The result compares numbers of weights and examples as both become large.

function varianceByRatio(ratios) {
  const variance = [];
  for (const ratio of ratios) {
    if (!(ratio > 0) || ratio === 1) {
      throw new Error('Ratio must be positive and different from 1');
    }
    if (ratio < 1) variance.push(ratio / (1 - ratio));
    else variance.push(1 / (ratio - 1));
  }
  return { ratios, variance };
}
const result = varianceByRatio([0.5, 0.75, 1.5, 3]);

The illustration has invented mistake counts for different sizes; move the control and read the plotted value. It is not experimental evidence or a prediction for your own network.

The paper analyzes particular linear-regression settings where such behavior can be derived mathematically. Whether more parameters help depends on the input distribution, correlations and the learning algorithm. Bigger does not always mean better.