The Modern Mathematics of Deep Learningexplained for non-mathematicians

Concepts

Learning foundations

Learning selects a prediction rule using examples. The example data below are invented.

Lessons

  1. Ridge regression
  2. Squared loss
  3. Empirical risk
  4. ReLU activation
  5. Gradient descent step
  6. Counting network parameters
  7. Sign classifier
  8. Two-unit ReLU network
  9. A fully connected network
  10. Empirical risk minimization
  11. Logistic surrogate loss
  12. Population risk
  13. Distance on a sample
  14. A mini-batch gradient step
  15. Rademacher complexity on a finite example
  16. Growth on a finite input space
  17. A weighted generalization bound
  18. Learning as a map
  19. Network parameter collection
  20. Bias as an extra coordinate
  21. A network hypothesis class
  22. Sign classifiers
  23. Mean surrogate loss
  24. Regularized objective minimum
  25. Classification risk
  26. Bayes risk on a finite space
  27. Regression risk decomposition
  28. Classification risk decomposition
  29. A no-free-lunch comparison
  30. A learned model
  31. An SGD convergence bound
  32. A union over network widths
  33. Fit every training point
  34. A matrix near the identity
  35. A Sobolev approximation bound
  36. Approximation risk and function distance
  37. VC dimension on two inputs
  38. Network VC bound factor
  39. VC generalization bound factor
  40. A nonzero linear functional
  41. Activation functions
  42. Hidden features and interpolation
  43. Testing an integral condition
  44. A second divided difference
  45. Uniform approximation on a compact set
  46. Covering a finite function class
  47. Three generalization scales
  48. A finite symmetrization example
  49. An entropy integral
  50. Covering classifiers and their losses
  51. Decomposing excess risk
  52. An interval indicator integral
  53. A nonconvex training objective
  54. An unbiased stochastic update
  55. Features and targets from a transformation
  56. Losses for density estimation
  57. The optimal binary label
  58. The source figure’s network
  59. A scaled sigmoid approaches a step
  60. Population risk after training
  61. Surrogate scores and classifiers
  62. A finite learning task
  63. A product-distributed sample
  64. Averaging optimization iterates
  65. A finite class of scores and labels
  66. A signed-measure functional and Fourier test
  67. A universal approximation condition
  68. A bounded squared loss
  69. Risk on a separate test sample
  70. A high-dimensional size scale