Hands-On Linear Regression Training with PyTorch

September 12, 2025

Based on Jupyter Notebook examples from my own experiments, this post uses PyTorch to build, train, and understand a linear regression model from scratch. It is aimed at beginners, and also helps you form an accurate mental model of PyTorch's autograd and training loop.

1. Recap: the linear regression problem

Linear regression is the most basic supervised learning model. Its mathematical form is:

y^=wx+b\hat{y} = w x + b

where:

  • xx: input feature
  • ww: weight
  • bb: bias
  • y^\hat{y}: predicted value

The goal of training is to learn suitable ww, bb from the data so that the prediction y^\hat{y} is as close as possible to the true value yy.

2. PyTorch basics: Tensors and autograd

What is a Tensor

In a Jupyter Notebook environment, install torch first, then create a sample tensor:

Creating a simple tensor

This is the simplest kind of Tensor: it has no gradient and is constructed from a matrix-like array. A Tensor has several important attributes, including shape, dtype, and device.

  • shape: the shape of the matrix, i.e. how many rows and columns.
  • dtype: the data type of the elements.
  • device: the device the tensor lives on, typically cpu or cuda.

A device of cuda enables NVIDIA's CUDA, running computation on the GPU, which can significantly speed up tensor operations.

Example:

Tensor shape, dtype and device

So a Tensor is essentially a wrapper around matrix data. Let's explore its other features.

Autograd on Tensors

Having looked at creating Tensors and their attributes, next comes a parameter you will use in any ML/LLM work: requires_grad. It is the core of creating tensors that support automatic differentiation:

Tensor with requires_grad

requires_grad means that computations involving the Tensor are tracked, so that you can later differentiate with respect to it (compute gradients).

In the example below, we first run a series of computations on Tensors:

Computations on tensors

You can see that the Tensors a, b, c, and d all have a grad_fn attribute, which indicates the operation that produced that Tensor.

Backpropagation on Tensors

Tensor backpropagation provides the ability to trace computations, but backward() can only be called directly on a scalar.

Gradient of a scalar

As shown above, the scalar x goes through a series of computations, and in the end its gradient (slope) can be read from the .grad attribute.

Besides what is covered here, Tensors also support median, max, sum, and much more; see the official documentation for details.

3. Building a linear layer with torch.nn.Linear

Here we use the linear model from torch's Neural Network module; the core is building a basic linear layer:

python
linear_layer = torch.nn.Linear(in_features=1, out_features=1)

print(linear_layer.weight)
print(linear_layer.bias)

This step is equivalent to defining the formula:

y=wx+by = wx + b

where:

  • weight corresponds to ww
  • bias corresponds to bb

4. Defining the linear regression model

Defining the linear regression model takes several steps.

Step 1: Prepare the data

The data here generates X in the formula with random numbers, then applies the basic computation plus a bit of random noise to simulate a real-world environment.

Generating regression data

Here true_W and true_b are the ground-truth values and y_true is the simulated output. Through training, we want to find W and b close to the ground truth.

In a real scenario, X might be an instrument's input (the independent variable), and y_true the historically recorded results for each X (the dependent variable). This data is what the model is trained on, and in practice it would be much larger.

Step 2: Define the regression model

There is more than one way to implement this. Although defining a model by subclassing torch.nn.Module is generally recommended, to illustrate what happens inside, I use a functional style here and compute the parameter updates by hand.

Functional regression training loop

Just this one loop is all it takes to train the model.

5. Forward pass

What is the forward pass

The forward pass feeds input data through the model to produce predictions.

wasm
input data → model → predicted result

(That is: input data → model → predictions.)

The model here starts out inaccurate, so its outputs may be wrong; the parameters need further adjustment to improve accuracy.

The corresponding code

Here is the code:

python
# forward pass and loss
y_hat = X @ W + b

PyTorch automatically builds the computation graph here and records the relationships between operations (see the Tensor section above).

6. Defining the loss function

The most common loss for linear regression is mean squared error (MSE):

python
loss = torch.mean((y_hat - y_true) ** 2)

Mathematically:

MSE=1N∑(y−y^)2\text{MSE} = \frac{1}{N} \sum (y - \hat{y})^2

The loss measures how good the model's predictions are; the smaller, the better.

7. Backward pass

What is the backward pass

During the forward pass, a computation graph is built and the relationships between operations are recorded. The backward pass propagates in the opposite direction along that graph, computing the gradients of every parameter with requires_grad=True.

The corresponding code

The backward pass is just one line:

python
# backward pass
loss.backward()

With this single call, we can compute the gradients of our two parameters, W and b.

8. Updating the parameters

The core snippet for updating parameters:

python
# update parameters
with torch.no_grad():
    W -= learning_rate * W.grad
    b -= learning_rate * b.grad

# zero gradients
W.grad.zero_()
b.grad.zero_()

Here we nudge the parameters to reduce the loss above, so that W and b approach their ground-truth values.

Two things to note:

  • So that adjusting W and b does not get recorded in the computation graph, we use torch.no_grad; inside it, tensor operations are not recorded in the graph.
  • After updating the parameters, we need to reset the gradients of W and b; otherwise they accumulate into the next iteration and cause confusion.

9. A more idiomatic approach

In real-world settings, training code is not usually written like this. With two parameters you can do it by hand, but LLMs have 7B, 256B, or more parameters, so writing it this way is completely impractical. torch already provides ready-made implementations of the manual computation above, and much more besides. Below is a more general approach (defining the model by subclassing torch.nn.Module and using torch's built-in machinery).

Regression model with nn.Module

Because torch already defines the common models, optimizers, and loss functions, following this conventional style hides all of the parameter-update logic inside the optimizer, which is very clean.

10. Summary

Through the hands-on example in this post, we started from the most basic mathematical model, walked through the entire process of training a linear regression model in PyTorch, and gradually opened up the "black box" of PyTorch's autograd and training mechanics.

The key takeaways:

  1. Linear regression is not complicated at its core Whether it is the mathematical formula

    y=wx+by = wx + b

    or PyTorch's nn.Linear, both are essentially an engineered wrapper around this one formula. Understanding that helps you keep your "model intuition" as you move on to more complex models.

  2. The Tensor is PyTorch's core data structure A Tensor is more than a "multi-dimensional array": with requires_grad=True, it becomes a node in the computation graph and automatically participates in gradient computation, which is the foundation of PyTorch's efficient training.

  3. Forward and backward form the training loop

    • Forward pass: compute predictions and loss from the current parameters
    • Backward pass: compute gradients by propagating the loss backward
    • Parameter update: use the gradients to update the parameters

    This flow applies not only to linear regression but to almost every deep learning model.

  4. Understanding a hand-written training loop matters Although in practice we usually use:

    • nn.Module
    • torch.optim
    • nn.MSELoss

    implementing parameter updates and gradient zeroing by hand lets you truly understand:

    • where gradients come from
    • why you need zero_grad()
    • why parameter updates belong inside torch.no_grad()
  5. High-level abstractions are built on low-level principles PyTorch's high-level APIs are not "magic"; they are a high-level abstraction over these basic steps. Once you understand the underlying logic, the standard style turns out to be both elegant and reliable.

Overall, linear regression is the best starting point for learning PyTorch and deep learning. If you truly understand every step in this post, then multi-layer networks, CNNs, RNNs, and today's popular large language models (LLMs) all share the same training lineage.

Directions you can try next:

  • Use real datasets (e.g. house prices, time series)
  • Introduce Dataset / DataLoader
  • Move the model to the GPU (CUDA)
  • Extend to multi-dimensional features or multi-layer networks

Understand the fundamentals, and you can go much further.