The Modern Mathematics of Deep Learningexplained for non-mathematicians

Concepts

Start here

These small examples introduce the operations used throughout the book. The data are invented.

Lessons

  1. Numbers and arrays
  2. Sum and mean
  3. Dot product
  4. Vector length
  5. Matrix times vector
  6. A changing output
  7. Number sets
  8. Indices from 1 to N
  9. Vector p-norm
  10. Frobenius norm
  11. Set membership
  12. Subsets
  13. Real vectors
  14. Real numbers
  15. Positive integers
  16. Index ranges
  17. Positive real values
  18. The unit interval
  19. Functions and domains
  20. Nonnegative functions
  21. Comparison up to a constant
  22. Distance and balls
  23. Matrix operator norm
  24. Measurable function space
  25. Random variables and their laws
  26. Expectation and variance
  27. Indicator functions
  28. Identity matrix
  29. Identity function
  30. Training samples
  31. Network parameters
  32. Binary labels
  33. Matrix dimensions
  34. Nonnegative real values
  35. Range of p
  36. Function Lp norm
  37. Sobolev spaces
  38. Outer product and transpose
  39. Distinct indices
  40. Natural logarithm
  41. Fourier transform and finite samples
  42. Inputs, targets and data
  43. Hypothesis sets
  44. A sample of examples
  45. A learning rule
  46. Activation functions
  47. Network architecture
  48. Independent random sample
  49. Multiple positive dimensions
  50. Image dimensions
  51. Kronecker delta
  52. Matrix shapes
  53. Indices outside an initial range
  54. Unit ball in Euclidean space
  55. Two index variables
  56. Coordinate of a vector-valued function
  57. A power function
  58. Closed intervals
  59. Vector and scalar domains
  60. Indices in a finite range
  61. Binary classification loss
  62. Projecting inputs and targets
  63. Allowed parameter vectors
  64. Layer weight and bias shapes
  65. Network width
  66. Continuous functions
  67. Empirical minimizers
  68. Training input indices
  69. Optimization iterates
  70. Mini-batch size
  71. Finite supremum
  72. Convex combination weights
  73. All-ones vector
  74. Vector-valued data samples
  75. Network diagram values
  76. Linear activation
  77. Exponential loss
  78. A linear prediction rule
  79. Binary vector space
  80. Polynomial feature vector
  81. Coordinate basis vector
  82. Finite measure and L2
  83. Function infinity norm
  84. Function norms with a measure
  85. Vector and scalar product space
  86. Bounded losses
  87. Positive constant pairs
  88. An input and target random pair
  89. Conditional expectation
  90. Values indexed by a set
  91. Multiple signal channels
  92. A batch of vectors
  93. Noisy observations
  94. Output-weight 1-norm
  95. First-layer neuron parameters
  96. One added input coordinate
  97. A small positive error
  98. Uniform random sign
  99. Finite range with a fixed bound
  100. Network depth condition
  101. Surprisal loss
  102. Hidden-layer sizes
  103. Index set difference
  104. A manifold and a chart
  105. A tangent space
  106. Partition of unity
  107. A function on a manifold
  108. Random feature parameters
  109. Initialization variances
  110. Random initial parameters
  111. Neural tangent kernel at parameters
  112. A radial target function
  113. Spaces in an inverse problem
  114. Signed measures
  115. Continuous function space and dual
  116. Drift and diffusion coefficients
  117. Wavefunction and density
  118. Electron coordinates
  119. Atomic numbers
  120. Figure axis ticks
  121. Widths used in a figure
  122. Approximation precision
  123. Unit vectors
  124. Compact parameter set
  125. Covering numbers
  126. A learned rule from random data
  127. Smooth functions
  128. An affine layer
  129. Parameter gradients
  130. Classification margin
  131. Trajectory figure architectures
  132. Sample sizes in source figures
  133. Larger image dimensions
  134. Polynomial degrees
  135. A zero output weight
  136. Zero hidden biases
  137. Radial directions and exponent
  138. Random sample pairs
  139. Linear regression with noise
  140. Linear classifier prediction
  141. A threshold feature
  142. Random feature probability measure
  143. Stochastic process path
  144. Initial data for the PDE
  145. Position and time
  146. A Lipschitz function
  147. Positive scale parameters
  148. Curvature index and loss
  149. Noise in the polynomial example
  150. Uniform input in the polynomial example
  151. Different samples and risks
  152. An optimization objective
  153. Short filter versus signal
  154. Matching residual dimensions
  155. Residual network input
  156. A PDE solution and initial condition
  157. A supervised loss pair
  158. Union of index ranges
  159. Mini-batch epoch count
  160. Zero initialization
  161. Bounded update direction
  162. A limiting kernel
  163. An empirical feature matrix
  164. Gaussian weight and bias initialization
  165. Network hypothesis class
  166. Layer indices and depth
  167. Dimensions and counts
  168. Positive theorem constants
  169. Target and approximating functions
  170. Weighted function error
  171. Fourier modulus descriptor
  172. Deformations and derivatives
  173. A finite Hilbert space example
  174. Batch mean and standard deviation
  175. Parameter-set boundary
  176. A network at selected parameters
  177. Feature operator parameters
  178. Finite function-class cardinality
  179. Gradient update directions
  180. A probability density estimate
  181. Square root of a density
  182. Composing the tent function
  183. Trajectory initialization scales
  184. Network input curves
  185. Binary classification losses
  186. Network input dimension
  187. A constant step schedule
  188. Smooth activation functions
  189. Distinct input points
  190. A separating parameter
  191. Parameter robustness radius
  192. Matrix pseudoinverse
  193. Unit noise second moment
  194. Training and independent test samples
  195. A class of loss functions
  196. Fourier transforms of products
  197. Neuron pre-activation
  198. ReLU nondifferentiability
  199. Approximation count scaling
  200. Smoothed margin
  201. A Sobolev function example
  202. A lower bound on k
  203. Sigmoid limits
  204. A fixed-input parameter map
  205. Zero target parameter
  206. Zero output weights
  207. Specific layer widths
  208. Affine region count factors
  209. Mapping inputs to basis vectors
  210. A compact training domain
  211. A single network layer
  212. Two-point source sample
  213. A six-entry parameter vector
  214. Parameters and depth product
  215. Gradient directions across steps
  216. A noisy target function
  217. Frame sizes
  218. Frame vectors
  219. Minimum batch size
  220. Pruning width and retained count
  221. Random iterative approximation
  222. Inverse square-root factor
  223. Manifold chart neighborhoods
  224. Localized chart functions
  225. A prior on a hypothesis set
  226. Source network parameter count
  227. A rotation matrix
  228. Equal random coordinates
  229. A single training example
  230. Activation properties in a theorem
  231. A complex Fourier phase
  232. Bounded target values
  233. A one-dimensional input
  234. Polynomial and activation equality
  235. Dimension-dependent approximation factors
  236. Rank-to-sample ratio
  237. Dimension-to-sample ratio
  238. Depth-separation size conditions
  239. Growing parameter norms
  240. Prediction and label in logistic loss
  241. Positive-spin wavefunctions
  242. Bound on the first sequence value
  243. A nonzero activation
  244. Interpolation parameter-count factors
  245. Parameters versus examples
  246. A Bayes-optimal predictor
  247. The marked risk identity
  248. hp finite element methods
  249. Balls and volume in one dimension