Depth and expressivity
Depth can change how efficiently a network represents certain functions. The examples do not prove a depth-separation theorem.
Lessons
- A function on the unit interval
- Composing ReLU layers
- A ReLU tent function
- Size for smooth-function approximation
- A depth separation statement
- Interpolation with repeated tent functions
- A radial function
- A hyperplane
- A shallow ReLU network
- A depth-width condition
- Weighted and Fourier errors
- A sum of ridge functions
- Convolution and support
- Where a unit switches
- Length of a feature trajectory
- Expected feature-curve length
- A density from Fourier energy
- A Sobolev approximation error
- Fourier transform of a ridge function
- The printed initialization scaling
- A hyperplane region bound
- Approximation size scales
- Checking size hypotheses
- A Sobolev norm hypothesis
- A product of hidden widths
- Directions in the source figure
- A two-dimensional network trajectory