Concept library
Optional explanations and runnable JavaScript examples for 506 concepts. The book does not require coding.
How the concepts connect
One penguin project connects the learning flow; tools and theory branch from it.
Learning flow
Collect examples Prepare inputs Make predictions Score mistakes Adjust the rule
Known targets also go to the objective. Updated parameters return to the rule. Evaluation applies the same chosen rule to separate new data. Check new examples
Tools, theory and applications
Math tools · Understand limits · Apply to science
Follow the shared worked example →
Collect examples
Record inputs and known answers or describe how examples are sampled.
- Feeds: Prepare inputs
- Feeds: Make predictions
- Feeds: Score mistakes
Prepare inputs
Turn inputs into features or signal representations.
- Feeds: Make predictions
Features · Frames and sparse signals · Signal transformations
Make predictions
Describe rules and parameters that produce outputs.
- Feeds: Score mistakes
- Feeds: Check new examples
Prediction rules and layers · Convolution · Batch normalization · Recurrent updates · Residual networks · Pruning
Score mistakes
Define losses and criteria for comparing candidate rules.
- Feeds: Adjust the rule
Adjust the rule
Select a rule or update its parameters.
- Feeds: Make predictions
Check new examples
Assess predictions using examples or a probability law.
Math tools
Use mathematical operations and notation.
- Supports: Make predictions
- Supports: Adjust the rule
- Supports: Score mistakes
Sets and notation · Vectors and matrices · Norms and distances · Probability and expectation · Functions and derivatives
Understand limits
Study statements, assumptions and bounds.
- Explains: Make predictions
- Explains: Adjust the rule
- Explains: Check new examples
Generalization and risk bounds · Approximation and depth · Optimization theory · Kernels and linearization · Source assumptions and notation · Model classes and structure · Local structure
Apply to science
Describe scientific problems and mathematical models.
- Supports: Make predictions
- Supports: Score mistakes
Inverse problems · PDE and stochastic processes · Quantum models
Browse by question or use
Observations and sampling
Describe examples, targets and samples.
- Inputs or assumptions
- Observations or a sampling law
- Outputs or conclusions
- Examples, labels or samples
- Training samples
- Binary labels
- Inputs, targets and data
- A sample of examples
- Independent random sample
- Training input indices
- Vector-valued data samples
- An input and target random pair
- A batch of vectors
- Noisy observations
- Random sample pairs
- Training and independent test samples
- A single training example
- A product-distributed sample
- Features and targets from a transformation
- Projecting inputs and targets
- One added input coordinate
- Linear regression with noise
- Two-point source sample
- A noisy target function
- Distinct input points
- Bounded target values
Features
Transform inputs into features used by a rule.
- Inputs or assumptions
- Inputs and a feature map
- Outputs or conclusions
- Feature values
Comes from
Used by
- Polynomial feature vector
- A threshold feature
- A sampled threshold feature
- Fixed features
- A feature is an output-weight derivative
- Multiple signal channels
- Feature operator parameters
- Mapping inputs to basis vectors
- Bias as an extra coordinate
- Hidden features and interpolation
- Directions in the source figure
- Duplicated hidden features
- Rescaled pruning feature
- Random feature parameters
Frames and sparse signals
Encode or reconstruct signals with frames and dictionaries.
- Inputs or assumptions
- Signals, frames or dictionaries
- Outputs or conclusions
- Coefficients or reconstructed signals
Related groups
- Supports: Features
- Circular window operation
- Hankel trace and reconstruction
- Frame encoder and decoder
- Dual-frame matrix identity
- Diagonal encoder coefficients
- Circular frame reconstruction
- Window and circular correlation
- Count nonzero entries
- Dictionary representation
- Thresholding coefficients
- Convolutional dictionary
- Local sparsity
- Sparse hierarchy
- Composed threshold maps
- Dictionary projection
- Thresholded coefficient approximation
- Frame sizes
- Frame vectors
- Convolution and cross-correlation
- A low-rank window matrix
- Check the printed threshold condition
- A multichannel signal window
- Local kernel support
- Short filter versus signal
Signal transformations
Study signal transforms, scattering and their properties.
- Inputs or assumptions
- Signals, transforms and assumptions
- Outputs or conclusions
- Transformed signals or statements about them
Related groups
- Supports: Features
- Fourier transform and finite samples
- Fourier modulus descriptor
- Deformations and derivatives
- Fourier transforms of products
- Translation and Fourier magnitude
- Convolution and modulus cascade
- Scattering feature norm
- Shift and Fourier phase
- Fourier energy preservation
- Deforming an input
- A zero-deformation comparison
Prediction rules and layers
Describe or apply prediction rules, layers and parameters.
- Inputs or assumptions
- Inputs, parameters and a rule
- Outputs or conclusions
- Predictions, layer values or model descriptors
- A linear prediction rule
- An affine layer
- Linear activation
- Activation functions
- Network parameters
- Layer weight and bias shapes
- ReLU activation
- Sign classifier
- Two-unit ReLU network
- A fully connected network
- Activation functions
- The source figure’s network
- A shallow network
- First-layer neuron parameters
- A zero output weight
- Zero hidden biases
- Residual network input
- A network at selected parameters
- Composing the tent function
- Network input dimension
- Smooth activation functions
- Neuron pre-activation
- A fixed-input parameter map
- Specific layer widths
- A single network layer
- A six-entry parameter vector
- Polynomial and activation equality
- A nonzero activation
- Network parameter collection
- Sign classifiers
- A learned model
- The optimal binary label
- Surrogate scores and classifiers
- Linearizing a network
- Composing ReLU layers
- Interpolation with repeated tent functions
- A shallow ReLU network
- A sum of ridge functions
- Where a unit switches
- A two-dimensional network trajectory
- Replacing unused hidden weights
- Two-layer ReLU network
- ReLU can pass a shifted input
- Conditional expectation
- A Bayes-optimal predictor
Convolution
Describe shared filters and downsampling.
- Inputs or assumptions
- Signals, filters and grid settings
- Outputs or conclusions
- Transformed signals or layer descriptions
Related groups
- Supports: Prediction rules and layers
Batch normalization
Describe batch normalization and its derivatives.
- Inputs or assumptions
- Batch values and normalization parameters
- Outputs or conclusions
- Normalized values or derivative relations
Related groups
- Supports: Prediction rules and layers
Recurrent updates
Update a state using successive inputs.
- Inputs or assumptions
- Previous state and next input
- Outputs or conclusions
- Next state
Related groups
- Supports: Prediction rules and layers
Residual networks
Describe residual layer updates and their continuous analogue.
- Inputs or assumptions
- Layer states, parameters or evolution assumptions
- Outputs or conclusions
- Layer states or evolution relations
Related groups
- Supports: Prediction rules and layers
Pruning
Describe removal of network weights and related error bounds.
- Inputs or assumptions
- A network, selection or bound assumptions
- Outputs or conclusions
- A pruned network or an error bound
Related groups
- Supports: Prediction rules and layers
Individual losses
Give one observed example a loss score.
- Inputs or assumptions
- A prediction or density, and an observed answer or example
- Outputs or conclusions
- A loss value
- Binary classification loss
- Exponential loss
- Surprisal loss
- Squared loss
- Logistic surrogate loss
- Losses for density estimation
- Margin surrogate loss
- Loss of a prediction rule
- A supervised loss pair
- Binary classification losses
- Prediction and label in logistic loss
- A bounded squared loss
- Linear classifier prediction
Objectives
Combine losses and penalties into a criterion for parameters or rules.
- Inputs or assumptions
- Losses, parameters and criterion settings
- Outputs or conclusions
- A scalar criterion
Comes from
Learning and parameter updates
Select a rule or update parameters using a learning rule.
- Inputs or assumptions
- Training examples or an objective, plus a learning or update rule
- Outputs or conclusions
- Selected prediction rules or updated parameters
Comes from
Used by
- Optimization iterates
- Gradient update directions
- A constant step schedule
- Gradient descent step
- A mini-batch gradient step
- Empirical risk minimization
- Regularized objective minimum
- An unbiased stochastic update
- Averaging optimization iterates
- Empirical minimizers
- Mini-batch size
- Mini-batch epoch count
- Gradient directions across steps
- Learning as a map
- Training starts at initialization
- A learned rule from random data
Initialization
Describe initial parameters and their distributions.
- Inputs or assumptions
- Architecture and initialization settings
- Outputs or conclusions
- Initial values or descriptions of their distribution
Risk and assessment
Distinguish loss on a sample from expected loss.
- Inputs or assumptions
- Predictions or a rule, and evaluation examples or probabilities
- Outputs or conclusions
- Risk, sample loss or classification margins
Sets and notation
Describe domains, indices and membership.
- Inputs or assumptions
- Sets and symbols
- Outputs or conclusions
- Membership and index relations
Related groups
- Supports: Model classes and structure
- Supports: Observations and sampling
- Number sets
- Indices from 1 to N
- Set membership
- Subsets
- Real vectors
- Real numbers
- Positive integers
- Index ranges
- Positive real values
- The unit interval
- Nonnegative real values
- Range of p
- Distinct indices
- Multiple positive dimensions
- Two index variables
- Closed intervals
- Indices in a finite range
- Indicator functions
- Kronecker delta
- Indices outside an initial range
- Unit ball in Euclidean space
- Vector and scalar domains
- Binary vector space
- Vector and scalar product space
- Union of index ranges
- A lower bound on k
- Dimensions and counts
- Balls and volume in one dimension
- Input and output grid groups
- An index map between grids
- Cyclic index representatives
Vectors and matrices
Combine numbers using vector and matrix operations.
- Inputs or assumptions
- Numbers, vectors or matrices
- Outputs or conclusions
- Computed vectors or scalars
Related groups
- Supports: Prediction rules and layers
- Supports: Objectives
- Supports: Learning and parameter updates
- Numbers and arrays
- Sum and mean
- Dot product
- Matrix times vector
- Identity matrix
- Matrix dimensions
- Outer product and transpose
- Matrix shapes
- All-ones vector
- Coordinate basis vector
- Values indexed by a set
- Matrix pseudoinverse
- Sample average
- Natural logarithm
- Unit vectors
- An empirical feature matrix
- A rotation matrix
- A one-sample matrix pseudoinverse
- A product of hidden widths
- Grid size
- A limit behind convergence
- A matrix near the identity
Norms and distances
Measure size or difference in a specified space.
- Inputs or assumptions
- Vectors, functions or a metric
- Outputs or conclusions
- A norm or distance
Related groups
- Supports: Objectives
- Supports: Approximation and depth
Probability and expectation
Describe random variables and their averages under a law.
- Inputs or assumptions
- A probability law and random variables
- Outputs or conclusions
- Probabilities, expectations or variance
Related groups
- Supports: Observations and sampling
- Supports: Risk and assessment
- Supports: Generalization and risk bounds
- Random variables and their laws
- Expectation and variance
- Finite measure and L2
- Uniform random sign
- Unit noise second moment
- A function defined by expectation
- Expectation of a threshold feature
- Signed measures
- Random feature probability measure
- Noise in the polynomial example
- Uniform input in the polynomial example
- Equal random coordinates
- Probability expression
- A probability density estimate
- A prior on a hypothesis set
Functions and derivatives
Describe maps and how their outputs change.
- Inputs or assumptions
- A function and its arguments
- Outputs or conclusions
- Values or derivatives
Related groups
- Supports: Prediction rules and layers
- Supports: Learning and parameter updates
- Functions and domains
- Nonnegative functions
- Identity function
- Coordinate of a vector-valued function
- A power function
- Continuous functions
- A changing output
- Parameter gradients
- Backpropagation through two layers
- Derivative of a product
- Product across layers
- Measurable function space
- Sobolev spaces
- Smooth functions
- A Lipschitz function
- ReLU nondifferentiability
- A Sobolev function example
- Sigmoid limits
- A complex Fourier phase
- An interval indicator integral
- A scaled sigmoid approaches a step
- ReLU and positive scaling
- A ReLU derivative identity
- A function on the unit interval
- A ReLU tent function
- A radial function
- A hyperplane
- A function on a subset
- Smooth local functions
- Support of a local weight
- A Fourier frequency bound
- Gradient change and curvature
- Second derivative blocks
- Shrinking and growing products
- A nonzero activation derivative
- Network input curves
Generalization and risk bounds
Relate sample behavior to population risk under stated assumptions.
- Inputs or assumptions
- A function class, sampling assumptions and bound parameters
- Outputs or conclusions
- Risk comparisons or bounds
Related groups
- Explains: Risk and assessment
- Explains: Objectives
- A weighted generalization bound
- Rademacher complexity on a finite example
- Growth on a finite input space
- VC dimension on two inputs
- Network VC bound factor
- VC generalization bound factor
- Covering a finite function class
- Three generalization scales
- A finite symmetrization example
- An entropy integral
- A complexity bound
- A margin-bound example
- Covering numbers
- A class of loss functions
- Regression risk decomposition
- Classification risk decomposition
- A no-free-lunch comparison
- Approximation risk and function distance
- Covering classifiers and their losses
- Decomposing excess risk
- Rademacher contraction example
- Peeling norm bounds
- A limiting variance curve
- Effective dimension and variance
- Variance with correlated coordinates
- The printed bias–variance equality
- A boundary case in the printed margin inequality
- An algorithm’s sample complexity measure
- Rank-to-sample ratio
- Dimension-to-sample ratio
- Bounded losses
Approximation and depth
Study approximation error and network size under stated conditions.
- Inputs or assumptions
- Target functions, network classes and assumptions
- Outputs or conclusions
- Approximation statements or bounds
Related groups
- Explains: Model classes and structure
- Explains: Prediction rules and layers
- A Sobolev approximation bound
- Uniform approximation on a compact set
- A universal approximation condition
- Size for smooth-function approximation
- A depth separation statement
- A depth-width condition
- A Sobolev approximation error
- Approximation size scales
- Barron approximation bound
- A sampling error bound
- Target and approximating functions
- Approximation count scaling
- Dimension-dependent approximation factors
- Depth-separation size conditions
- Interpolation parameter-count factors
- A nonzero linear functional
- Testing an integral condition
- A second divided difference
- A signed-measure functional and Fourier test
- Weighted and Fourier errors
- Convolution and support
- Length of a feature trajectory
- Expected feature-curve length
- A density from Fourier energy
- Fourier transform of a ridge function
- A hyperplane region bound
- Checking size hypotheses
- A Sobolev norm hypothesis
- Combining local descriptions
- Fourier representation and sampling weights
- A problem in the printed representation
- Approximating a random quantity by steps
- Feature approximation error
- Finite approximation inequality
- A radial target function
- Radial directions and exponent
- Square root of a density
- Continuous function space and dual
- Approximation precision
- A small positive error
- A compact training domain
- Inverse square-root factor
- A high-dimensional size scale
Optimization theory
Study convergence and loss geometry under assumptions.
- Inputs or assumptions
- An objective, update rule and assumptions
- Outputs or conclusions
- Convergence or geometry statements
Related groups
- Explains: Learning and parameter updates
- Explains: Initialization
- An SGD convergence bound
- A nonconvex training objective
- Smallest eigenvalue bound
- Geometric loss decrease
- A gradient upper comparison
- Bounded update direction
- A separating parameter
- Parameter robustness radius
- Smoothed margin
- Growing parameter norms
- Exponential loss and margin
- A supremum equality
- Balanced adjacent layers
- Exponential risk and its gradient
- Projected normalized gradient
- Duplicate features and interpolation
- Output-weight gradient and kernel
- Feature change after parameter movement
- Risk along a weight interpolation
- Two connected parameter paths
- Staying near initialization
- Expected Hessian operator norm
- Parameters along a low-loss construction
- Training time and width factors
Kernels and linearization
Relate parameter gradients or features to kernel matrices.
- Inputs or assumptions
- Features or parameter gradients
- Outputs or conclusions
- Kernel values or matrix relations
Related groups
- Supports: Optimization theory
Source assumptions and notation
Interpret symbols and assumptions in a source statement.
- Inputs or assumptions
- Symbols and assumptions in a source statement
- Outputs or conclusions
- Interpretation of the statement, not model data
- Comparison up to a constant
- Positive constant pairs
- Finite range with a fixed bound
- Index set difference
- Figure axis ticks
- Sample sizes in source figures
- Positive scale parameters
- Curvature index and loss
- Positive theorem constants
- Zero target parameter
- Affine region count factors
- Activation properties in a theorem
- Bound on the first sequence value
- The marked risk identity
- An undefined source symbol
Model classes and structure
Describe families of rules, their size and structural properties.
- Inputs or assumptions
- A rule family and structural assumptions
- Outputs or conclusions
- Class descriptions, counts or structural statements
- Hypothesis sets
- A learning rule
- Network architecture
- Allowed parameter vectors
- Network width
- Finite supremum
- Convex combination weights
- Compact parameter set
- Hidden-layer sizes
- Network hypothesis class
- Parameter-set boundary
- Finite function-class cardinality
- Parameters and depth product
- Parameters versus examples
- Counting network parameters
- A network hypothesis class
- A union over network widths
- A finite learning task
- A finite class of scores and labels
- A norm-constrained network class
- The printed initialization scaling
- Image dimensions
- Larger image dimensions
- Network diagram values
- Widths used in a figure
- Network depth condition
- Trajectory figure architectures
- Polynomial degrees
- Layer indices and depth
- Source network parameter count
- A one-dimensional input
Local structure
Describe charts, local coordinates and localization.
- Inputs or assumptions
- A domain, charts and assumptions
- Outputs or conclusions
- Local descriptions or structural statements
Related groups
- Supports: Approximation and depth
Inverse problems
Describe reconstruction from observations using a forward operator.
- Inputs or assumptions
- Observations, a forward operator and a candidate
- Outputs or conclusions
- A reconstruction or mismatch criterion
Related groups
- Supports: Objectives
- Supports: Prediction rules and layers
PDE and stochastic processes
Describe differential equations and stochastic representations.
- Inputs or assumptions
- Equations, initial data and process assumptions
- Outputs or conclusions
- Solutions, samples or mathematical relations
Related groups
- Supports: Approximation and depth
- Random iterative approximation
- Sampling a PDE representation
- A one-dimensional PDE example
- A deterministic process in a PDE representation
- Random input for a PDE example
- Drift and diffusion coefficients
- Stochastic process path
- Initial data for the PDE
- Position and time
- A PDE solution and initial condition
- hp finite element methods
Quantum models
Describe wavefunctions, particle coordinates and energy relations.
- Inputs or assumptions
- Particle configurations, wavefunctions and operators
- Outputs or conclusions
- Features, energies or mathematical relations
Related groups
- Supports: Prediction rules and layers
- Eigenvalue equation
- Hamiltonian potential terms
- Eigenvalue equation
- Swapping particles
- Finite Rayleigh quotient and local energy
- Spin-block determinants
- Permutation-equivariant features
- Distance-based embeddings
- Determinants with learned factors
- Embedding into a feature network
- A product of orbitals
- Particle and nucleus coordinates
- Wavefunction input and output
- Separate spin-group embeddings
- Wavefunction and density
- Electron coordinates
- Atomic numbers
- Positive-spin wavefunctions
Source formulas · Glossary · JavaScript basics
An independent educational explanation of The Modern Mathematics of Deep Learning — Julius Berner, Philipp Grohs, Gitta Kutyniok, Philipp Petersen. Examples use invented numbers unless a measured dataset is credited.