Designing a network

Nearby pixels can form a small part of an edge. A filter is a short list of weights applied to a small group of nearby brightness readings. It multiplies each reading by its matching weight and adds the results.

Slide the same two-number filter (1, −1) across toy values (3, 2, 4). Multiply matching entries and add in each window.

function neighboringDifferences(values) {
  return { outputs: [values[0] - values[1], values[1] - values[2]] };
}
const result = neighboringDifferences([3, 2, 4]);

Slide the same filter [1, −1] across readings [3, 2, 4]. In the first position compute 1×3 − 1×2 = 1; in the second compute 1×2 − 1×4 = −2. The filter is reused instead of learning a new pair of weights at every position.

This repeated local calculation is the idea behind convolutional networks. A later stage can summarize nearby filter responses and reduce the picture’s size; this is often called pooling. The paper studies how local calculations, shared weights and behavior when a pattern shifts restrict which functions a network can express.

During training, an adjustment signal travels backward through the layers. Each layer multiplies it by a factor describing that layer’s effect. For example, a signal of 1 multiplied by 0.5 three times becomes 0.5, then 0.25, then 0.125. Many factors below 1 can make it tiny; many above 1 can make it huge. Either can make early layers hard to adjust.

A residual block keeps the earlier value and adds a correction, instead of completely replacing it.

In the paper’s residual layer, the activated correction is added to the previous layer. As a worked example, use previous value 3 and correction 1.

function residualBlock(value, correction) {
  const activated = Math.max(0, correction);
  return { input: value, correction: activated, output: value + activated };
}
const result = residualBlock(3, 1);

Here the barred Φ is the residual layer output, ℓ is its layer number, Φ is the computed correction before activation, ρ is the activation, and θ contains the parameters. In this example ReLU keeps 1, so 3 + 1 = 4.

The direct path also lets an adjustment signal pass backward without going through every correction. The paper relates many small updates to describing motion by calculating it one small time step after another. A shortcut can help, though it does not guarantee easy training.

When we shrink a picture, we can lose tiny but important details. A U-shaped network first compresses information, then builds a picture back up. Its long connections can bring earlier detail to the reconstruction.

Average two toy pixel values 2 and 4, and keep their half-difference. Use both pieces to reconstruct the inputs.

function splitAndReconstruct(first, second) {
  const average = (first + second) / 2;
  const detail = (first - second) / 2;
  const reconstructed = [average + detail, average - detail];
  return { average, detail, reconstructed };
}
const result = splitAndReconstruct(2, 4);

From pixels 2 and 4, the average is (2 + 4)/2 = 3 and the half-difference is (2 − 4)/2 = −1. Keeping both lets us recover 3 − 1 = 2 and 3 + 1 = 4. The average alone would produce two 3s and forget the contrast.

The average describes what the two pixels share; the difference describes how they contrast. The paper uses a richer collection of local and global patterns, called a frame, to describe an image with numbers saying how much of each pattern is present. Under stated conditions those numbers can be used to reconstruct the image.

In a U-Net the skips carry useful earlier features across the compression path. Whether reconstruction is exact depends on its filters and architecture, not merely on drawing a U shape.

Different training batches can send a layer numbers with different centers and spreads. Batch normalization changes their scale using the values in the current batch, then applies learnable scale and shift parameters.

Take a batch containing 1 and 3. First find its average; then measure spread by squaring each distance from that average, averaging those squares and taking the square root.

function normalizeBatch(values) {
  if (values.length === 0) throw new Error('Batch cannot be empty');
  let total = 0;
  for (const value of values) total += value;
  const mean = total / values.length;
  let squaredTotal = 0;
  for (const value of values) squaredTotal += (value - mean) ** 2;
  const spread = Math.sqrt(squaredTotal / values.length);
  if (spread === 0) throw new Error('Batch must have nonzero spread');
  const normalized = [];
  for (const value of values) normalized.push((value - mean) / spread);
  return { mean, spread, normalized };
}
const result = normalizeBatch([1, 3]);

The average is (1 + 3)/2 = 2. Subtract it from each reading to get differences −1 and +1. Square them, then average: ((−1)² + 1²)/2 = 1. Its square root is 1, the spread called standard deviation. Subtract 2 and divide by this spread to get [−1, 1].

The paper studies how this normalization can change how loss responds when settings move. Under certain conditions it can make that response smoother, helping the search for settings. It does not guarantee a better final model.

If every number in a batch is identical, the standard deviation is zero. The bare formula cannot divide by zero; practical implementations add a small stabilizing term. The illustration uses nonidentical values.

Big networks can contain connections that add little to a particular output. Pruning sets some weights to zero so fewer connections need to be kept.

A toy output adds three weights times inputs. Remove a small weight, then compare the resulting output to the original.

function pruneWeights(weights, cutoff) {
  if (!(cutoff >= 0)) throw new Error('Cutoff must be nonnegative');
  let original = 0;
  let pruned = 0;
  for (const weight of weights) {
    original += weight;
    if (Math.abs(weight) >= cutoff) pruned += weight;
  }
  return { original, pruned };
}
const result = pruneWeights([2, 0.1, 1], 0.2);

For a toy input with all features equal to 1, contributions 2, 0.1 and 1 add to 3.1. Removing the 0.1 contribution gives 3, only 0.1 away for this input. But other inputs could change the effect of removing a connection.

The paper proves that under specified assumptions a very wide two-layer ReLU network can be approximated by a much sparser one when some retained output weights are adjusted. The guarantee concerns closeness to the ORIGINAL network’s output, not automatic correctness against true labels.

Its illustration of sparse connections is about removing weights. It is different from the sparse pattern descriptions we will meet in chapter 7.

A photo is a single collection of pixels. A sentence or a series of bell rings comes in an order. Sometimes the next answer should depend on what came earlier.

A recurrent network keeps a state: a number or collection of numbers carrying information from earlier inputs. It updates that state when the next input arrives, using the same rule each time.

For a different sequence from the activity, imagine a bell that gives 1 whenever it rings. Each step halves the previous state and adds the current ring.

function recurrentStep(first, memory, nextInput) {
  const second = memory * first + nextInput;
  return { first, second };
}
const result = recurrentStep(1, 0.5, 1);

Start with state 0. If a bell ring adds 1 and the old state is halved, the first step gives 0/2 + 1 = 1. At the second ring it gives 1/2 + 1 = 1.5. The number 1.5 carries information from both steps.

To train the repeated rule, the network sends an adjustment signal backward through the steps. Repeated multiplication can make that signal tiny or huge, making earlier steps hard to adjust. These are vanishing and exploding gradients. Some designs use gates to control what the state keeps; the state is not perfect memory.