Introduction

Intro to Neural Networks and Backpropagation (2h 25m):

Intro to Language Modeling (1h 57m):

Building makemore Part 2: MLP (1h 15m):

Building makemore Part 3: Activations & Gradients, BatchNorm (1h 55m):

Building makemore Part 4: Becoming a Backprop Ninja (1h 55m):

Building makemore Part 5: Building a WaveNet (56m):

Let's build GPT: from scratch, in code (1h 56m):

Let's build the GPT Tokenizer (2h 13m):

• Backpropagation: The engine implements backpropagation.
• Dynamically Built DAG: The computations are represented as a Directed Acyclic Graph (DAG) that is built dynamically.
• Small Neural Networks Library: On top of the autograd engine, there is a small neural network library that mimics the API of PyTorch.
• Scalar Operations: The DAG operates on scalar values, meaning each computation is broken down into the smallest possible operations (additions and multiplications). This granularity helps in understanding how neural networks operate at a fundamental level.
• Educational Purpose: The simplicity and small size of the code make it ideal for learning and teaching purposes. Despite its simplicity, it is powerful enough to construct and train deep neural networks for tasks like binary classification.

Chain Rule in the Context of Forward and Backward Passes

Understanding the Chain Rule in Multivariable Calculus

The General Chain Rule

Scalar-Valued Functions

Addition

Multiplication

Topological Sort:

Role of Topological Sort in Backpropagation

Example Workflow

1a = 2
2b = 3
3c = a * b  # c = 6
4d = c + b  # d = 9
5e = d + a  # e = 11

Explanation

Initialization:

build_topo Function

• This is a recursive function that performs a depth-first traversal of the graph.
• If a node v has not been visited, it is marked as visited.
• The function recursively visits all the nodes in v._prev (i.e., the parent nodes of v), exploring as far down each branch as possible before backtracking.
• After visiting all its parent nodes, v is appended to the topo list.

Topological Sort

Backward Pass

• The gradient of the final output node (self) is initialized to 1.
• The nodes are processed in reverse topological order, starting from the output node and moving backward through the graph.
• The _backward method for each child node is called to propagate its gradient to its parents according to the chain rule.

Double Underscore Functions in Python

Power Function (__pow__)

1def __pow__(self, other):
2    assert isinstance(other, (int, float)), "only supporting int/float powers for now"
3    out = Value(self.data**other, (self,), f'**{other}')
4
5    def _backward():
6        self.grad += (other * self.data**(other-1)) * out.grad
7    out._backward = _backward
8
9    return out

Explanation:

Purpose

Parameters

• self: The current Value object.
• other: The exponent, which must be an integer or float.

Forward Pass

• The data attribute of self is raised to the power of other.
• A new Value object, out, is created with the result of the exponentiation. This new object keeps track of self as its parent, and the operation performed (exponentiation) is recorded.

Backward Pass

• The _backward function calculates the gradient of the exponentiation operation using the chain rule.
• The partial derivative is calculated according to the power rule of the derivative which is given below.
• This derivative is multiplied with out.grad to propagate the gradient backward.

Return

ReLU Function (relu)

1def relu(self):
2    out = Value(0 if self.data < 0 else self.data, (self,), 'ReLU')
3
4    def _backward():
5        self.grad += (out.data > 0) * out.grad
6    out._backward = _backward
7
8    return out

Explanation:

Purpose

Mathematical Definition

• If the input x is greater than 0, the output is x.
• If the input x is less than or equal to 0, the output is 0.

Forward Pass

• The data attribute of self is compared to 0.
• If self.data is less than 0, out.data is set to 0.
• Otherwise, out.data is set to self.data.
• A new Value object, out, is created to store the result of the ReLU operation, keeping track of self as its parent and the operation performed (ReLU).

Backward Pass

• The _backward function calculates the gradient of the ReLU operation using the chain rule.
• The derivative of ReLU is 1 if self.data is greater than 0, and 0 otherwise.
• This derivative is multiplied by out.grad to propagate the gradient backward.

Return

• The function returns the new Value object out that represents the result of the ReLU activation.

1def __neg__(self):
2    return self * -1

• Purpose: This method allows the unary negation operator (-) to be used with Value objects. This function will help us to tackle subtraction as a special case of addition. It is used in __sub__ method of value class.
• Explanation: When -self is called, it returns the result of multiplying the Value object by -1. This effectively negates the value.

1def __radd__(self, other):
2    return self + other

• Purpose: This method allows the addition operator (+) to be used with Value objects on the right-hand side.
• Explanation: __add__ method which is defined above is forself+ other.When other + self is called and other does not have its own __add__ method that can handle the addition, Python falls back to __radd__. This method simply calls self + other, leveraging the existing __add__ method.

1def __sub__(self, other):
2    return self + (-other)

• Purpose: This method allows the subtraction operator (-) to be used with Value objects.
• Explanation: When self - other is called, it returns the result of adding self to the negation of other. This is done using the previously defined __neg__ method.

1def __rmul__(self, other):
2    return self * other

• Purpose: This method allows the multiplication operator (*) to be used with Value objects on the right-hand side.
• Explanation: When other * self is called and other does not have its own __mul__ method that can handle the multiplication, Python falls back to __rmul__. This method simply calls self * other, leveraging the existing __mul__ method.

1def __truediv__(self, other):
2    return self * other**-1

• Purpose: This method allows the true division operator (/) to be used with Value objects.
• Explanation: When self / other is called, it returns the result of multiplying self by the reciprocal of other. It handles division as a special case of multiplication. This uses the exponentiation operator (**) to raise other to the power of -1.

1def __rtruediv__(self, other):
2    return other * self**-1

• Purpose: This method allows the true division operator (/) to be used with Value objects on the right-hand side.
• Explanation: When other / self is called and other does not have its own __truediv__ method that can handle the division, Python falls back to __rtruediv__. This method returns the result of multiplying other by the reciprocal of self.

1def __repr__(self):
2    return f"Value(data={self.data}, grad={self.grad})"

• Purpose: This method provides a string representation of the Value object.
• Explanation: When repr(self) or print(self) is called, it returns a string that includes the data and grad attributes of the Value object. This is useful for debugging and logging purposes.

Why __radd__ and __rmul__ Are Needed

Example Scenario

Without __radd__

• Python first tries to call the __add__ method on the integer 3, but the integer class does not know how to handle a Value object.
• This results in a TypeError.

With __radd__

• When Python cannot find a suitable __add__ method on the left operand (the integer 3), it looks for an __radd__ method on the right operand (the Value object).
• The __radd__ method on the Value object is called, successfully handling the operation and returning the correct result.

Example

• c = a+b
• d = a×b
• f = c+d

Forward Pass

1a = Value(2.0)
2b = Value(3.0)

Backward Pass

1e.grad = 1

For e:

• e._backward() propagates e.grad (which is 1) to c and d.
• c.grad += e.grad → c.grad = 1
• d.grad += e.grad → d.grad = 1

For d:

• d._backward() propagates d.grad to a and b.
• a.grad += b.data * d.grad → a.grad += 3.0 * 1 → a.grad = 3
• b.grad += a.data * d.grad → b.grad += 2.0 * 1 → b.grad = 2

For c:

• c._backward() propagates c.grad to a and b
• a.grad += c.grad → a.grad += 1 → a.grad = 4
• b.grad += c.grad → b.grad += 1 → b.grad = 3

nn.py