Practice / Initialization and optimizers
Xavier and He initialization: the variance by hand
Before you start
Before the first gradient step, a network's weights are random numbers, and their variance decides whether a signal that enters the first layer arrives at the last one at a usable size. Too small and every layer shrinks it, so the output and the gradients vanish; too large and they explode. The two standard answers, Xavier (Glorot and Bengio, 2010) and He (He et al., 2015), come from the same short calculation: the variance of a sum of products of independent random numbers. These ten problems do that calculation forwards and backwards, through a ReLU and a leaky ReLU, for convolutions, for uniform distributions and through residual stacks. The five mistakes at the end each change the variance by a constant factor per layer, which depth turns into an exponential: a variance used where a second moment belongs, Xavier trusted on a ReLU network, a uniform's half-width read as its standard deviation, a convolution's fan-in counted as its channels, and a residual block assumed to preserve what its branch preserves.
- is expectation and . The second moment equals the variance exactly when . If and are independent, , and the same holds for any functions of them, such as and .
- A dense layer is with , so . The fan-in is the number of inputs each output sums; the fan-out is the number of outputs each input feeds. Biases are initialized to and left out.
- At initialization the are independent and identically distributed, independent of the layer's input, with mean and variance , and symmetric: has the same distribution as . (normal) and (uniform on ) are both symmetric.
- A random variable is symmetric about if has the same distribution as ; then . Every distribution on this page is continuous, so .
- , and is for and for . The leaky ReLU with slope is for and for . Both act entry by entry.
- A deep network: and for , with the input, and weight variance . Entries of one layer are treated as identically distributed, so means the variance of any one entry.
- The backward pass uses the code-style names of the minibatch page: is for a scalar loss . Through , , so ; through , , with the elementwise product.
- All of this describes the network at initialization, averaged over the random weights. The backward calculations also treat a layer's as independent of the arriving from above, the simplification Glorot and Bengio made; it is not exact, but it predicts the scale well.
Builds on: One-hidden-layer backprop, the whole backward pass
Problems
- ·
Let and be independent with and . Show that . Evaluate it for and an with mean and variance , and compare it with .
- ··
A dense layer has weights as in Before you start, and inputs with a common second moment ; the need not be independent of each other. Show that , and find the that makes .
- ··
Let be symmetric about with variance . Show that . For , also find and .
- ···
In the deep network of Before you start, suppose the entries of are symmetric about . Show that , that the entries of are again symmetric, and find the that keeps the variance fixed.
- ··
A ReLU network of constant width uses in every layer. Find . Evaluate it for and , which is Xavier's choice when , and say what does over the same depth.
- ··
In the backward pass through , . Suppose the share a second moment and are independent of . Show that . Explain Xavier's , and evaluate it and both scale factors for a layer from inputs to outputs.
- ···
In a ReLU network with He's fan-in initialization , the gradient goes back through and . Treat as independent of and as independent of . Show that , and find the factor from back to for widths , , , . Which variance would make each factor ?
- ·
A 2-D convolution has input channels, output channels, a kernel and stride : . Find its fan-in and fan-out, and the He and Xavier weight variances.
- ···
For a leaky ReLU, show that for symmetric about , and find the weight standard deviation that preserves the variance, where is the gain. Show that and find the uniform bound with the same variance. PyTorch's
nn.Linearinitializes its weight withkaiming_uniform_(weight, a=math.sqrt(5)), whereais the slope that enters the gain: what are its bound and variance? - ···
A residual stack computes for , with a scalar . Each branch is initialized so that and (for example with ). With , find . Evaluate it for , , and for with ; show that the second is below for every .
Answers
- , while
- , so keeps
- ; for normal , and
- ; He initialization makes the factor exactly
- ; for , it is , a standard deviation times smaller
- ; for , Xavier gives , a forward factor and a backward factor
- ; from layer back to layer the factor is ; (fan-out) makes each factor
- fan-in , fan-out ; He (standard deviation ); Xavier
- , , uniform bound ; with slope the bound is and the variance
- : for , , and for ,
Worked solutions
Problem 1
Let and be independent with and . Show that . Evaluate it for and an with mean and variance , and compare it with .
- .Independence factorizes the expectation of the product, and kills it whatever 's mean is.
- .The definition of variance with step 1's mean; and are functions of independent variables, so they are independent too.
- .The definition of variance rearranged, with a zero mean.
- .The same identity for , whose mean is not zero, so its square stays.
- , while The product of the variances misses : because 's sign is random, even a constant input contributes spread to . The two agree only when , which a ReLU output never satisfies (Problem 3).
Problem 2
A dense layer has weights as in Before you start, and inputs with a common second moment ; the need not be independent of each other. Show that , and find the that makes .
- .Linearity of expectation, then independence of from , then the zero mean of the weights.
- .Square the sum and use linearity; there are terms.
- For : ., and the input are mutually independent. The cross terms vanish because of the weights' zero means, so correlated inputs do no harm.
- For : .Problem 1, step 2, with .
- , so keeps Only the diagonal terms survive, and with a zero mean the second moment is the variance. This is LeCun's initialization. It preserves the scale of a linear network, and of a tanh network near , where . For it is a standard deviation of .
Problem 3
Let be symmetric about with variance . Show that . For , also find and .
- when and when . passes positive values and zeroes the rest.
- , where is the expectation of times the indicator of the event . has the same distribution as , and swaps the two events while leaving unchanged.
- .The two events cover everything except , where ; and , so the second moment is the variance.
- .Steps 1 to 3: the left side is one of two equal halves of .
- For : .The integrand is the normal density times on , and is an antiderivative of .
- ; for normal , and The variance is step 4 minus step 5 squared. The next layer needs the second moment, , not this variance (Problem 2), and the is what He's factor undoes (Problem 4).
Problem 4
In the deep network of Before you start, suppose the entries of are symmetric about . Show that , that the entries of are again symmetric, and find the that keeps the variance fixed.
- .Problem 2 with , which is computed from earlier layers only and so is independent of .
- .Problem 3 applied to , which is symmetric by hypothesis.
- .Substitute step 2 into step 1.
- Negating row of negates and leaves the joint distribution of the weights and unchanged, so is symmetric.The weights are symmetric and independent of everything below them. The first layer's output is symmetric for the same reason, whatever the input, so the hypothesis holds at every layer from on.
- ; He initialization makes the factor exactly The standard deviation is , which is for . PyTorch's
kaiming_normal_withnonlinearity='relu'uses this, as with gain ; its default mode is'fan_in'.
Problem 5
A ReLU network of constant width uses in every layer. Find . Evaluate it for and , which is Xavier's choice when , and say what does over the same depth.
- for .Problem 4, step 3, with .
- .Apply step 1 for : that is factors. The first layer reads the raw input, not a ReLU output, so it is not one of them.
- For , : ., so .
- For : .The same formula with .
- ; for , it is , a standard deviation times smallerThe standard deviation scales by the square root, . A constant factor per layer becomes exponential in depth, so only neither vanishes nor explodes; too much variance per layer already multiplies it by over twenty layers.
Problem 6
In the backward pass through , . Suppose the share a second moment and are independent of . Show that . Explain Xavier's , and evaluate it and both scale factors for a layer from inputs to outputs.
- .Entry of combines column of with : one term per output.
- and .Problem 2's argument with the roles swapped: the cross terms vanish for because and are independent with mean , and the sum now runs over terms.
- The forward pass keeps its scale when (Problem 2), the backward pass when .Both hold only if .
- gives .Glorot and Bengio's compromise: the variance is the reciprocal of the average fan, so the two factors average to .
- ; for , Xavier gives , a forward factor and a backward factor Neither direction is preserved exactly; each is off by the ratio of one fan to the average. The derivation assumes linear units, so on a ReLU network both factors are halved again (Problems 5 and 7).
Problem 7
In a ReLU network with He's fan-in initialization , the gradient goes back through and . Treat as independent of and as independent of . Show that , and find the factor from back to for widths , , , . Which variance would make each factor ?
- is with probability and otherwise. is symmetric about (Problem 4, step 4) and continuous, so it is positive half the time.
- .The two factors are treated as independent, and , , so the square of the mask is the mask.
- .Problem 6, step 2: has rows, so the backward sum has terms.
- .Steps 2 and 3 with : the ReLU's cancels the .
- From to : .Step 4 for ; the product telescopes.
- ; from layer back to layer the factor is ; (fan-out) makes each factor With fan-in initialization the backward pass is off only by the ratio of two widths, not by an exponential in depth, which is why the fan-in mode is a safe default. PyTorch exposes the other choice as
mode='fan_out'.
Problem 8
A 2-D convolution has input channels, output channels, a kernel and stride : . Find its fan-in and fan-out, and the He and Xavier weight variances.
- Each output is a sum over and : products.Fan-in counts the products summed into one output, and a convolution sums over every input channel and every kernel position.
- .Problem 2 with : the weights in one output's sum are distinct and independent. The same weights are reused at other positions, which correlates different outputs but does not change the variance of one.
- An input away from the border appears in outputs, each through a different weight.It sits at each of the kernel offsets of some window, for each of the output channels; the backward sum of Problem 6 has that many terms.
- fan-in , fan-out ; He (standard deviation ); Xavier Both are channels times the receptive-field size , which is how PyTorch computes the fans of a convolution weight.
Problem 9
For a leaky ReLU, show that for symmetric about , and find the weight standard deviation that preserves the variance, where is the gain. Show that and find the uniform bound with the same variance. PyTorch's nn.Linear initializes its weight with kaiming_uniform_(weight, a=math.sqrt(5)), where a is the slope that enters the gain: what are its bound and variance?
- for and for .The leaky ReLU multiplies negative inputs by , so their squares by .
- .Problem 3, steps 2 and 3: each sign carries half of .
- , so and .Problem 4, step 3, with step 2 in place of the ReLU's . gives He's and , a linear unit, gives LeCun's .
- .The density is on the interval and the mean is ; .
- gives .Match the uniform's variance to the normal one's.
- Slope in step 3's gain: , so .
kaiming_uniform_computes its bound as gain times , with the leaky-ReLU gain of step 3 evaluated at itsa; a slope above is not a sensible activation, only a way to set the scale. - , , uniform bound ; with slope the bound is and the variance That variance is a sixth of He's , so a plain stack of default
nn.Linearlayers with ReLUs shrinks the variance by per layer (Problem 5 with ). The same step 5 gives He uniform and Xavier uniform .
Problem 10
A residual stack computes for , with a scalar . Each branch is initialized so that and (for example with ). With , find . Evaluate it for , , and for with ; show that the second is below for every .
- , with and .Expand the square and use linearity of expectation.
- .The cross term is and , by the hypotheses on the branch.
- .Apply step 2 for each of the blocks.
- , : .The skip and the branch each carry the full second moment, and they add.
- : , which for is .Step 3 with .
- , so . for , because is concave and is its tangent line at .
- : for , , and for , Uncorrelated signals add their second moments, so a stack of variance-preserving branches still grows exponentially with depth. Scaling the branches by caps the growth at however deep the stack; some ResNet recipes instead start each branch's last layer at zero, which is at initialization.
Where this goes wrong
1. He variance computed from Var(h) instead of E[h²]
Initialization is described as keeping "the variance" fixed, so it is natural to put the previous layer's variance into the formula.
- Right so far: Problem 4, step 1, for width .
- “For independent factors, , so the input enters through its variance.”The shortcut that causes the mistake: that product rule needs both means to be , and a ReLU output has mean (Problem 3).
- , so The second moment, , is what reaches the next layer (Problem 1), giving . With each layer multiplies the variance by about , and twenty layers by about .
2. Xavier assumed to keep a ReLU network's variance constant
Xavier initialization is a common library default (Keras's Dense layer uses glorot_uniform), and it was derived to keep the forward variance fixed.
- when Right so far: Problem 6, step 4, at equal widths.
- “With , Problem 2 says each layer passes its variance on unchanged.”The shortcut that causes the mistake: Problem 2 preserves the second moment of the layer's input, and Xavier's derivation assumes that input is the previous layer's output unchanged, as in a linear network.
- A ReLU sits between the layers and halves the second moment (Problem 3), so the factor is per layer: after layers the variance is of the first layer's (Problem 5), and the gradients shrink the same way.
3. He uniform bound set to the standard deviation √(2/n_in)
The normal and the uniform versions of He initialization are both described by the scale , and that number gets used as the uniform's limit.
- He initialization wants Right so far: Problem 4.
- “Sample .”The shortcut that causes the mistake: treating a uniform's half-width as its standard deviation.
- (Problem 9), so this gives , a third of the target, and the variance falls by a factor of per ReLU layer. The bound must be times larger: .
4. Convolution fan-in counted as the input channels
For a dense layer the fan-in is the number of input features, and a convolution's input features look like its channels.
- , where the fan-in is the number of products summed into one outputRight so far: Problem 2.
- “The layer has input channels, so its fan-in is .”The analogy that causes the mistake: a convolution sums over the kernel's positions as well as the channels.
- Each output sums products (Problem 8), so with ReLUs each layer multiplies the variance by , the kernel area: five such layers multiply it by . The correct variance is .
5. Residual block assumed to preserve variance because its branch does
Each branch of a residual network is initialized to preserve the scale of its input, so the whole block looks as if it does too.
- and Right so far: the branch hypotheses of Problem 10.
- “The branch preserves the variance and the skip connection passes through untouched, so their sum keeps it too.”The analogy that causes the mistake: treating the sum of two signals of the same size as if it had that size.
- Uncorrelated terms add their second moments, so each block doubles it (Problem 10): over blocks, about for . Scale the branches by , or start them at zero.
Print this set: xavier-and-he-initialization.pdf (problems, answers, and worked solutions on separate pages).
More at eulerpad.app. This is one of 82 free practice sets with worked solutions at eulerpad.app/practice, alongside guides to the topics behind them.
EulerPad, the app, reads your handwritten math on iPad, iPhone and Android and checks it line by line. It's free to start.
This set is also Chapter 20 of Deep Learning Math by Hand, a 250-problem workbook with worked solutions: eulerpad.app/workbooks