PyTorch For Deep Learning — nn.Linear and nn.ReLU Explained

A Neural Network consist of Layers such as Linear and activation function like ReLU . let’s see what they are

What is nn.Linear ?

figure 1.1

as shown in figure 1.1, we know that each layer of a neural network in a feed forward neural network will have it’s own weight matrix and biases. (which represents the weights of each neuron in the layer)

The nn.Linear layer can be used to implement this matrix multiplication of input data with the weight matrix and addition of the bias term for each layer.

Example of nn.Linear

1. Importing the necessary libraries

import torch
import numpy as np
from torch import nn

2. Creating an object for linear class

linear_layer = nn.Linear(in_features=3,out_features=1)

This takes 2 parameters. input features and output features, which are the number of inputs and number of outputs.
This will create a weight matrix and bias vector randomly as shown in the figure 1.1.
size of the Weight matrix : 3x1
size of the Bias Vector : 1x1

3. To see the weights and biases

print(linear_layer.weight)
print(linear_layer.bias)
output:
tensor([[ 0.1806, -0.0349, -0.1638]], requires_grad=True)
tensor([-0.2685], requires_grad=True)

4. passing input to this linear layer

print(linear_layer(torch.tensor([1,2,3],dtype=torch.float32)))
print(0.1806*1 - 0.0349*2 - 0.1638*3 -0.2685)
output:
tensor([-0.6490], grad_fn=<AddBackward0>)
-0.6491

so, this input is multiplied with the weight matrix of the layer and bias is added as shown in figure 1.1.
the result is the number : -0.639

That’s it for linear layer

nn.ReLU

Relu is an activation function that is defined as this:
relu(x) = { 0 if x<0, x if x > 0}.

after each layer, an activation function needs to be applied so as to make the network non-linear. there are several other activation functions like relu

ReLU in PyTorch

relu = nn.ReLU()
print(relu(torch.tensor(50,dtype=torch.float32)))
print(relu(torch.tensor(-50,dtype=torch.float32)))
output: 
tensor(50.)
tensor(0.)

as show above in relu function’s definition, It keeps the positive numbers as the number itself and it for negative number, it returns 0.

Conclusion

nn.Linear is a function that takes the number of input and output features as parameters and prepares the necessary matrices for forward propagation.

nn.ReLU is used as an activation function to make the network non-linear and fit complex data