The Modern Mathematics of Deep Learningexplained for non-mathematicians

Concepts

Special architectures

These components constrain or organize computations in different ways.

Lessons

  1. Circular convolution
  2. Derivative of a product
  3. Batch normalization
  4. Residual layer
  5. Downsampling a grid
  6. Circular window operation
  7. Pruning weights
  8. A recurrent update
  9. A pruning error bound
  10. Convolution then downsampling
  11. Hankel trace and reconstruction
  12. Frame encoder and decoder
  13. Batch normalization Jacobian
  14. Pruned network construction
  15. Dual-frame matrix identity
  16. Diagonal encoder coefficients
  17. Batch normalized gradient norm
  18. Sparse approximation bound
  19. Circular frame reconstruction
  20. Product across layers
  21. Residual difference
  22. Continuous residual evolution
  23. Normalize a whole batch
  24. Window and circular correlation
  25. Rescaled pruning feature
  26. Two-layer ReLU network
  27. Finite approximation inequality
  28. Feature norm comparison
  29. ReLU can pass a shifted input
  30. Batch normalization as an identity
  31. Convolution layer as a matrix
  32. Zero-padded window convolution
  33. Input and output grid groups
  34. Two normalized batch directions
  35. Shrinking and growing products
  36. Convolution and cross-correlation
  37. A sequence of residual states
  38. A nonzero activation derivative
  39. An index map between grids
  40. A low-rank window matrix
  41. Pruning hypotheses on a finite network
  42. From input batch to normalization
  43. Cyclic index representatives