· 2 min read
Overview
This is my ongoing project of writing a transformer from scratch with no help from AI. I was inspired by this blog post to take on this challenge. I gathered a lot of my learning resources from that post and will be listing the ones that helped me the most here, along with notes for my own reference.
Understanding Backpropagation
https://youtu.be/i94OvYb6noo?si=e29xHU_vWdCl2HjB
Take-aways from this video
- We represent the process of backpropagation with a computation DAG, where the input side contains all input data values and weights of the neural network, and the output resolves to a scalar loss value. Each node in the graph represents a mathematical operation.
- In the forward pass, the input data and weights are passed through the DAG topologically (meaning every node of one layer must finish computing before the next layer begins) where the mathematical operations manipulate the values until they reach the final loss calculation node, where the computed values are compared to the input's assigned label values. The loss value is a representation of the difference between the DAG's understanding of the relationship between input and output vs the actual relationship.
- Backpropagation uses the loss value to determine the "effect" that a particular input value has on the loss calculation. This "effect" value, or gradient, is used in the optimization step to update the model's weights (input data values are not updated) in a direction that will ultimately reduce the loss value the next time the inputs are passed through the DAG.
- Backpropagation is a recursive process that moves topologically from output to input. For a given node, a gradient is computed for each input parameter by multiplying the input's partial derivative (which is the "local gradient") by the gradient from the node's output. To kick off the process, the loss node's output contains a gradient value of 1, because the partial derivative of L w.r.t L is an identity function.
- For nodes that have multiple outputs the gradient value is accumulated via summation.
- Typically, inputs to the computational graph nodes are tensors, meaning that backpropagation on a computational node would produce a Jacobian matrix; a 2D matrix that shows the influence of every element in the input on every element of the output of the node. The gradient calculation for the input tensor would then be the Jacobian matrix (of input parameter vs output aka the local gradient) multiplied by the gradient tensor of the output (effect of node's output on the loss calculation)
- It would be impractical to actually compute the full Jacobian due to its size, so in practice each backward gate's implementation instead hand-codes the effect of its sparse structure directly, without ever materializing the matrix.