Why add more layers?

Why would a network need several layers? Imagine a lamp in the middle of a circular garden. The light at a flower depends only on how far it is from the lamp, not whether it is left or right.

Consider the toy rule that returns distance from the origin. Points (3, 4) and (0, 5) lie at the same radius.

function distances(firstPoint, secondPoint) {
  const first = Math.sqrt(firstPoint[0] ** 2 + firstPoint[1] ** 2);
  const second = Math.sqrt(secondPoint[0] ** 2 + secondPoint[1] ** 2);
  return { first, second };
}
const result = distances([3, 4], [0, 5]);

For the point (3, 4), square and add: 3×3 + 4×4 = 25. Since 5×5 = 25, the square root is 5, its distance from (0, 0). The lamp activity uses a different center, at 5: flowers at 3 and 7 are both 2 units away, so they get the same light.

The paper constructs a special rule based on distance. A three-layer network can approximate it with manageable size. For a particular class of two-layer networks, comparable accuracy needs exponentially more hidden neurons: the required number can grow by repeated multiplication as the problem size grows.

This is a mathematical separation for a specially constructed target and distribution under the theorem’s assumptions. It does not say every deep network beats every shallow network.

ReLU uses only straight pieces: it keeps positive values and clips negative ones. Combining several ReLU terms can make a line rise, turn and fall, like a small tent.

Combine two shifted ReLUs in the paper’s hat function, defined for inputs from 0 to 1. Try 0, 0.5 and 1.

function tent(value) {
  return 2 * Math.max(0, value) - 4 * Math.max(0, value - 0.5);
}
function tentExamples(value, inputs) {
  const examples = [];
  for (const input of inputs) examples.push(tent(input));
  return { output: tent(value), examples };
}
const result = tentExamples(0.5, [0, 0.5, 1]);

Try x = 0.25. The first part gives 2×0.25 = 0.5; the second sees 0.25 − 0.5 = −0.25, so ReLU makes it 0. Thus h(0.25) = 0.5. At x = 0.75, the parts give 1.5 and 4×0.25 = 1, so h(0.75) = 0.5 too. Feed 0.5 back in: h(0.5) = 1. Repeating the rule can double the number of peaks.

The lamp illustration shows a similar rise and fall on a larger scale, from 0 to 4 with a peak at 2. The paper’s tent runs from 0 to 1 and peaks at 0.5. In the next activity, the unit-interval tent is fed through itself again.

That efficiency helps certain deep networks approximate curved functions, including multiplication, by piecing together many short straight segments. It is a statement about representing a function, not automatically about finding the weights during training.

We have measured networks by how closely they can match a target. Another way to measure their possibilities is to count how often the direction of their output curve can change.

Walk from 0 to 3, then back to 1: three steps forward and two back. Add both distances to measure the whole path, not just the distance between its ends.

function journey(middle, end) {
  const length = Math.abs(middle) + Math.abs(end - middle);
  return { length, netChange: end };
}
const result = journey(3, 1);

One tent has two straight segments. Passing its output through the tent again gives four; doing so once more gives eight. After k compositions this constructed rule has 2^k pieces; this does not count every function a k-layer network can produce.

Each ReLU neuron changes behavior when its weighted input crosses zero. Stacking layers can split and fold the space into many regions where the network follows different straight rules.

The paper also studies how a curve drawn in input space can stretch as it passes through random layers. Such results need specific initialization assumptions; seeing a busy curve in a drawing is not itself a theorem.