Finding useful patterns

Move a drawn cat a little to the right. For a cat-or-dog question we probably want the answer to stay the same. A useful feature can ignore an irrelevant change while still responding to important changes.

Imagine an arrow pointing right. Turn it downward without changing its length: its direction changes, but its length stays the same. A complex number can be pictured as such an arrow. A Fourier coefficient uses one to describe the strength and alignment of a repeating pattern in a signal. Frequency tells how many times the pattern repeats around a loop. Try a signal sampled at four positions around a loop, with one mark; move the mark one position.

function shiftedMagnitude(shift) {
  const angle = -2 * Math.PI * shift / 4;
  return { first: 1, second: Math.hypot(Math.cos(angle), Math.sin(angle)) };
}
const result = shiftedMagnitude(1);

The coefficient changes from 1 to −i: its arrow turns from right to down. This arrow represents a number; it is not an arrow drawn on the image. Its length, or magnitude, stays 1; its direction, or phase, changes. The source formula gives a continuous shift identity; this finite circular example illustrates it but does not prove it. In the formula u is the signal, τ the shift and ω the frequency.

A scattering transform applies filters to find patterns, takes the magnitude of each response, then averages over an area. It can repeat these steps to describe patterns at different scales. Its theory gives guarantees for particular translations and small deformations, such as a slight stretch.

Invariance is “the feature stays the same”; stability is “a small input change causes only a small feature change.” The paper does not say every image change should be ignored, or that ordinary learned networks automatically satisfy the same guarantees.

Describe a house drawing using stored patterns such as a roof, walls and a door. A number for each pattern says how much of it to use. If only a few numbers are nonzero among many available patterns, the description is sparse.

The collection of patterns is called a dictionary. Here the simplest dictionary, the identity, matches each input entry to its own pattern, so the amounts stay [2, 0.1]. Now shrink them with a threshold of 0.2.

function thresholdCoefficients(values, cutoff) {
  const coefficients = [];
  for (const value of values) {
    coefficients.push(Math.max(0, value - cutoff) - Math.max(0, -value - cutoff));
  }
  return { coefficients };
}
const result = thresholdCoefficients([2, 0.1], 0.2);

For these positive amounts, subtract 0.2 from 2 to get 1.8; 0.1 is below the threshold, so it becomes 0. This is soft thresholding: it also shrinks amounts that remain. Negative amounts are handled by the other term in the formula. In the formula D is the dictionary, Dᵀx gives pattern amounts, b is the threshold and ρR is ReLU.

Thresholding changes the amounts used to describe patterns; pruning in chapter 6 removes network weights. The paper connects sparse pattern descriptions to convolutional networks: early layers can detect local patterns; further layers can combine them into more structured descriptions.

Thresholding can also remove a real but weak clue. Mathematical recovery guarantees need assumptions about how sparse the description is, how different the patterns are and how much noise was added. The point is to understand possible learned features, not to claim every network discovers this dictionary.