Python PyTorch Autograd: requires_grad and backward

python pytorch autograd requires_grad and backward requires a clear understanding of the core syntax, runtime behavior, and practical implementation patterns demonstrated in the examples below.

When you call backward() on a tensor in PyTorch, the autograd engine computes gradients for every tensor that has requires_grad set to True. This mechanism is the core of training neural networks, but it also introduces subtle behavior that can trip up even experienced developers. This article explains how requires_grad and backward work together, how to control gradient tracking, and where common mistakes occur.

How Autograd Builds a Computational Graph

PyTorch's autograd records operations on tensors that have requires_grad=True. Each operation creates a node in a directed acyclic graph (DAG) that connects the input tensors to the output. The graph stores the function used to compute the output, so that during backward() the gradient can be propagated back through the chain rule.

Consider a simple expression:

import torch

x = torch.tensor([2.0], requires_grad=True)
y = x * 3
z = y.sum()

Here, x is a leaf tensor because it was created directly by the user. y is a non-leaf tensor produced by the multiplication operation. z is the result of a sum. The autograd graph links z to y and y to x. When you call z.backward(), PyTorch walks this graph backward, computing gradients for each tensor that requires them.

Only tensors with requires_grad=True are included in the graph. If a tensor has requires_grad=False, operations on it are not recorded, and it cannot receive a gradient.

Setting requires_grad on Tensors

The requires_grad flag controls whether autograd tracks operations on a tensor. By default, tensors created with torch.tensor() have requires_grad=False. You can enable it at creation time:

x = torch.tensor([1.0, 2.0], requires_grad=True)

Or you can set it later using the .requires_grad_() method (note the trailing underscore, which indicates an in-place operation):

x = torch.ones(3)
x.requires_grad_(True)

When you set requires_grad=True on a tensor, PyTorch starts recording every operation that involves that tensor. This is necessary for training, but it also increases memory usage because the graph must be stored until backward() is called.

For tensors that are used only for inference or as fixed inputs, leaving requires_grad=False avoids unnecessary graph construction. You can also disable gradient tracking temporarily with torch.no_grad() or permanently with detach().

Calling backward() to Compute Gradients

The backward() method computes the gradient of the current tensor with respect to all tensors that require gradients. It is typically called on a scalar loss value:

loss = (y - target).pow(2).sum()
loss.backward()

After calling backward(), the .grad attribute of each leaf tensor with requires_grad=True is populated with the gradient. For example:

x = torch.tensor([2.0], requires_grad=True)
y = x ** 2
y.backward()
print(x.grad)  # tensor([4.0])

The gradient is accumulated into .grad. If you call backward() again without clearing the gradient, the new gradient is added to the existing value. This is intentional for gradient accumulation across mini-batches, but it can lead to unexpected results if you forget to zero the gradients before each training step.

For non-scalar tensors, you must provide a gradient argument that matches the shape of the tensor. This argument represents the gradient of the final loss with respect to that tensor. For example:

x = torch.tensor([1.0, 2.0], requires_grad=True)
y = x * 2
# y is a vector, so we need to pass a gradient vector
y.backward(torch.tensor([1.0, 1.0]))
print(x.grad)  # tensor([2.0, 2.0])

In practice, you rarely call backward() on a non-scalar tensor directly; instead, you reduce the output to a scalar loss first.

Stopping Gradient Tracking: detach() and no_grad()

There are two common ways to prevent autograd from tracking operations: detach() and torch.no_grad(). They serve different purposes.

tensor.detach() returns a new tensor that shares the same data but has requires_grad=False and is not connected to the current graph. This is useful when you need to use a tensor's value for something that should not affect gradient computation, such as logging or computing a metric:

x = torch.tensor([1.0], requires_grad=True)
y = x * 2
y_detached = y.detach()  # no gradient connection

torch.no_grad() is a context manager that disables gradient tracking for all operations inside the block. It is commonly used during evaluation or inference:

with torch.no_grad():
    predictions = model(x_val)

Inside the block, tensors created from operations have requires_grad=False, and existing tensors with requires_grad=True do not have their operations recorded. This saves memory and computation because no graph is built.

A related context manager is torch.inference_mode(), which is even more aggressive and can be faster for pure inference, but it disallows operations that would require gradient tracking. Use no_grad() when you need flexibility, and inference_mode() when you are certain no backward pass will ever be needed.

Common Pitfalls with requires_grad and backward

Several subtle issues arise when working with autograd. One is in-place operations on tensors that require gradients. For example, using x.add_(1) on a leaf tensor can corrupt the graph because the original value is overwritten, and autograd may raise an error or produce incorrect gradients. In general, avoid in-place operations on tensors that are part of the graph.

Another pitfall is forgetting to zero gradients before calling backward() again. Gradients accumulate by default, so if you train a model without calling optimizer.zero_grad(), the gradients from previous steps are added to the new ones, leading to unstable training.

Leaf tensors are also a common source of confusion. Only leaf tensors get their .grad populated by default. Non-leaf tensors do not store gradients unless you explicitly use retain_grad(). If you need the gradient of an intermediate tensor, you must set retain_grad() on it, or use register_hook.

Finally, calling backward() on a tensor that is not a scalar and without a gradient argument raises a runtime error. Always reduce the output to a scalar loss before calling backward().

Performance and Memory Implications of Autograd

Building the computational graph has a real cost. Every operation on a tensor with requires_grad=True allocates memory for the graph node and stores intermediate values needed for the backward pass. This can significantly increase memory usage compared to inference-only code.

To minimize overhead, disable gradient tracking whenever you do not need gradients. For example, when evaluating a model on a validation set, wrap the forward pass in torch.no_grad(). This prevents the graph from being built and reduces memory consumption.

Another consideration is the retain_graph parameter of backward(). By default, the graph is freed after a backward pass to save memory. If you need to call backward() multiple times on the same graph (e.g., for computing higher-order gradients), you must pass retain_graph=True. This keeps the graph alive but increases memory usage.

For large models, the graph can become the dominant memory consumer. Techniques like gradient checkpointing trade computation for memory by recomputing intermediate activations during the backward pass. However, for most use cases, simply using no_grad() during inference and clearing gradients each step is sufficient.

Practical Example: Training a Linear Regression Model

To see requires_grad and backward() in action, consider a minimal linear regression training loop. The model has one parameter w and one bias b. We create them with requires_grad=True so that autograd tracks operations on them.

import torch

# Training data
x = torch.linspace(0, 10, 100)
y_true = 3 * x + 2 + torch.randn(100) * 0.5

# Model parameters
w = torch.randn(1, requires_grad=True)
b = torch.zeros(1, requires_grad=True)

learning_rate = 0.01
for epoch in range(100):
    # Forward pass
    y_pred = w * x + b
    loss = (y_pred - y_true).pow(2).mean()

    # Backward pass
    loss.backward()

    # Update parameters (no autograd tracking)
    with torch.no_grad():
        w -= learning_rate * w.grad
        b -= learning_rate * b.grad

    # Zero gradients
    w.grad.zero_()
    b.grad.zero_()

print(f"Learned w: {w.item():.3f}, b: {b.item():.3f}")

In this loop, loss.backward() computes the gradients of w and b. The parameter update is done inside torch.no_grad() to avoid building a graph for the update operation itself. After updating, we zero the gradients so the next backward pass does not accumulate.

This example demonstrates the core pattern used in all PyTorch training: define parameters with requires_grad=True, compute a loss, call backward(), update parameters without gradient tracking, and zero gradients before the next iteration. Understanding requires_grad and backward() is essential for debugging and extending this pattern to more complex models.