Neural Networks from Zero: Embeddings and Backpropagation

Imagine a small network learning to guess the next musical note from the last few notes. It starts out terrible. Its parameters are random, so its predictions are nearly random too. Every time it is wrong, training nudges those parameters in a direction that would have made the correct note more likely.

Repeat that enough times and the network starts to learn the patterns in the training data.

This is a toy example, not a description of every neural network. It is just large enough to show the complete learning loop without hiding the machinery.

1. embeddings turn symbols into learned vectors

The network cannot do arithmetic directly on note names like a, b, or c. It uses an embedding table to map each note to a vector of numbers:

a -> [ 0.1, -0.3,  0.2, ...]
b -> [ 0.7, -0.1,  0.0, ...]
c -> [-0.2,  0.4,  0.8, ...]

Those numbers usually begin as random values and change during training. The individual dimensions are not hand-labeled traits. What matters is the learned geometry of the vectors: notes used in similar contexts can develop useful relationships in that space.

For this example, give each note a two-dimensional embedding.

2. context combines the recent notes

The model looks at the last three notes to predict the next one. Each note is a two-number vector, so concatenating the three embeddings produces one six-number context vector:

3 notes * 2 dimensions = 6 input values

Concatenation is a deliberately simple choice for this network. Larger sequence models use more powerful mechanisms, but the purpose is the same: turn the available context into numbers the next layer can process.

3. the hidden layer builds a useful representation

Each hidden unit receives the same six input values but has its own learned weights and bias. It computes an affine transformation:

z = inputs * weights + bias

Then it applies a nonlinear activation such as tanh. Without a nonlinearity, stacked linear layers would still collapse into one linear transformation. The activation lets the network represent relationships a straight line cannot.

If the hidden layer has ten units, it produces ten new values. They are not yet predictions. They are an intermediate representation that makes the final prediction easier to compute.

4. logits become probabilities

The output layer turns those ten hidden values into one score for every possible next note. These raw scores are the logits.

For seven possible notes, the model emits seven logits. Softmax converts them into probabilities that sum to one:

a: 0.45
b: 0.22
c: 0.01
...

If the correct note is a, cross-entropy loss is the negative log-probability assigned to a. A confident correct prediction has low loss. A confident wrong prediction has high loss.

5. backpropagation assigns credit and blame

Backpropagation computes how the loss would change if each parameter changed. That derivative is the parameter's gradient. Gradient descent then applies a small update:

parameter <- parameter - learning_rate * gradient

The update reaches every trainable part of this network: output weights, hidden weights, biases, and the embedding vectors themselves. No individual unit knows the right answer. The gradient connects the final error to every earlier choice that contributed to it.

6. generation feeds predictions back into the model

After training, the network can generate a sequence by repeatedly using its own output as new input:

start with a rest
predict e
feed e back in
predict f
feed f back in
continue until the model predicts another rest

In one sentence: recent notes become learned vectors, the network transforms that context into hidden features, logits score the possible next notes, cross-entropy measures the error, and backpropagation adjusts every parameter that helped produce it.