Layers and Activations
Layers transform data; activation functions add the non-linearity that makes networks powerful.
A network is a stack of layers. A dense (fully-connected) layer multiplies inputs by weights and adds a bias. Then an activation function bends the result.
Common activations
- ReLU —
max(0, x); the default for hidden layers. - Sigmoid — squashes to 0-1; for binary output.
- Softmax — turns scores into class probabilities that sum to 1.
Without a non-linear activation, stacking layers would collapse into a single linear model.
Example
import numpy as np
def relu(x): return np.maximum(0, x)
def sigmoid(x): return 1 / (1 + np.exp(-x))
x = np.array([-2.0, -0.5, 0.5, 2.0])
print(relu(x)) # [0. 0. 0.5 2. ]
print(np.round(sigmoid(x), 2)) # [0.12 0.38 0.62 0.88]When to use it
- A computer-vision engineer adds a Dropout layer after each Dense layer to prevent a deep image classifier from memorising training examples.
- An NLP practitioner stacks multiple Linear layers with ReLU activations to build a text classifier that can learn abstract token representations.
- A researcher replaces sigmoid activations with ReLU in a deep network to avoid the vanishing gradient problem and achieve faster convergence.
More examples
Linear layer with ReLU and Dropout
Builds a two-layer network with ReLU non-linearity and 30% Dropout regularisation, a standard pattern for avoiding overfitting.
import torch.nn as nn
model = nn.Sequential(
nn.Linear(128, 64),
nn.ReLU(),
nn.Dropout(p=0.3),
nn.Linear(64, 10)
)
print(model)Compare activation functions
Computes ReLU, Sigmoid, and Tanh activations on the same input values to compare how each squashes and clips the signal differently.
import torch
import torch.nn as nn
x = torch.linspace(-3, 3, 7)
relu = nn.ReLU()(x)
sigmoid = nn.Sigmoid()(x)
tanh = nn.Tanh()(x)
print('ReLU: ', relu.detach().round(decimals=2))
print('Sigmoid:', sigmoid.detach().round(decimals=2))
print('Tanh: ', tanh.detach().round(decimals=2))Custom layer by subclassing nn.Module
Implements a residual block by subclassing nn.Module and adding a skip connection (output = input + transformed input), the core idea behind ResNets.
import torch
import torch.nn as nn
class ResidualBlock(nn.Module):
def __init__(self, dim):
super().__init__()
self.fc = nn.Linear(dim, dim)
self.relu = nn.ReLU()
def forward(self, x):
return x + self.relu(self.fc(x)) # skip connection
block = ResidualBlock(64)
x = torch.randn(8, 64)
print(block(x).shape)
Discussion