Generalization
Performance on a training sample and performance on new examples are different quantities.
Lessons
- ReLU and positive scaling
- Minimum training margin
- Neural tangent kernel
- Margin surrogate loss
- A complexity bound
- Loss of a prediction rule
- Linearizing a network
- Gradient flow and the kernel
- Exponential loss and margin
- Minimum kernel norm interpolation
- A norm-constrained network class
- Rademacher contraction example
- A supremum equality
- Peeling norm bounds
- A margin-bound example
- A limiting variance curve
- Effective dimension and variance
- Variance with correlated coordinates
- Balanced adjacent layers
- A constant-kernel flow solution
- The printed bias–variance equality
- Training loss and a robust margin
- Exponential risk and its gradient
- Projected normalized gradient
- Backpropagation through two layers
- A boundary case in the printed margin inequality
- A one-sample matrix pseudoinverse
- An algorithm’s sample complexity measure
- A ReLU derivative identity
- A width limit and a kernel space