Optimization
The shape of an objective affects how training algorithms move.
Lessons
- Squared training objective
- A shallow network
- Fixed features
- Duplicate features and interpolation
- Features, Gram matrix and gradient
- Smallest eigenvalue bound
- Gradient change and curvature
- Geometric loss decrease
- Replacing unused hidden weights
- Output-weight gradient and kernel
- A feature is an output-weight derivative
- Feature change after parameter movement
- Duplicated hidden features
- Second derivative blocks
- Risk along a weight interpolation
- Two connected parameter paths
- Gaussian initialization density
- Staying near initialization
- Probability expression
- A limit behind convergence
- Expected feature Gram matrix
- Expected Hessian operator norm
- A gradient upper comparison
- Parameters along a low-loss construction
- Feature approximation error
- Training time and width factors
- The spin-glass parameter sphere
- Training starts at initialization
- Population risk of network parameters