Xavier, Kaiming, and Batch Normalization Explained

The first note followed a small network through its whole loop:

• notes become embedding vectors;
• a hidden layer transforms the context;
• an output layer produces logits;
• cross-entropy measures the error;
• backpropagation updates the parameters.

There is one problem hiding underneath that clean story: the scale of the numbers matters. A network can have the right architecture and still be hard to train because its activations or gradients become too large, too small, or too close to a flat part of the activation function.

1. large inputs can saturate tanh

A hidden unit first computes a weighted sum:

z = inputs * weights + bias
h = tanh(z)

If the initial weights make z very large in either direction, tanh(z) lands near -1 or 1. Those are the flat ends of the curve, where the derivative is close to zero. Backpropagation then sends a very small gradient through that unit, so learning slows down. This is saturation.

That is different from a dead ReLU. A ReLU outputs zero for negative inputs. A ReLU unit can become “dead” when it stays on that zero-gradient side for the data it sees. Saturated tanh units and dead ReLUs both create weak gradients, but they are not the same failure mode.

Weights that are too small can cause a different problem: signals and gradients can shrink as they move through many layers. Good initialization tries to keep their scale useful in both directions.

2. Xavier and Kaiming solve related, different problems

The shared goal is to keep the variance of activations and gradients from exploding or disappearing as they pass through the network. The right formula depends on the activation.

Xavier for linear or tanh-style layers

Xavier, or Glorot, initialization accounts for both the number of inputs (fan_in) and outputs (fan_out). For a normal distribution, its basic scale is:

standard deviation ~= sqrt(2 / (fan_in + fan_out))

An activation-specific gain can adjust that scale. Xavier is a common starting point for linear and tanh-style layers because it balances the signal flowing forward with the gradient flowing backward.

Kaiming for ReLU-style layers

Kaiming, or He, initialization compensates for the fact that ReLU discards the negative half of its input. In the common fan_in form for ReLU:

standard deviation ~= sqrt(2 / fan_in)

So 1 / sqrt(N) is not a universal rule called “Kaiming init.” Xavier and Kaiming are related variance-preserving strategies with different assumptions. The activation function determines which assumption fits.

3. batch normalization standardizes; it does not bound

Even with sensible initialization, activation distributions move while the weights are learning. Batch normalization makes that movement easier to manage.

For each feature in a mini-batch, it computes the batch mean and variance, then standardizes the activations:

z_norm = (z - batch_mean) / sqrt(batch_variance + epsilon)

Before the learned affine step, the normalized values have approximately zero mean and unit variance for that batch. They are not constrained to the range 0 to 1; negative values and values larger than one are normal.

The layer then learns a scale gamma and shift beta:

y = gamma * z_norm + beta

That gives the network room to choose a different distribution when the task benefits from one. During training, batch normalization uses the current mini-batch statistics. During inference, it uses running estimates accumulated during training.

4. batch statistics introduce noise

Every mini-batch contains a different sample of the data, so its mean and variance differ slightly:

batch 1 mean =  0.12
batch 2 mean =  0.05
batch 3 mean = -0.03

That variation moves the normalized activations, which moves the logits and the loss. The effect depends on batch size and composition: with very small batches, the statistics can be noisy enough to make training unstable; with larger, representative batches, they are usually steadier.

the practical picture

Initialization chooses a useful signal scale before training begins. Xavier is a natural default for the tanh network in this example; Kaiming is designed for ReLU-style activations. Batch normalization can then stabilize changing activation distributions during training without forcing them into a fixed range.

They solve different parts of the same problem: keeping information and gradients alive long enough for the network to learn.