Neural networks explained: neurons, activations, and layers

Prerequisites

Before diving in, make sure you are comfortable with:

• Logistic regression: a single neuron is essentially logistic regression with a nonlinear twist
• Matrix multiplication: the core operation in every neural network layer

Interactive visualization: Neural Network

A live 2→4→1 network learning XOR. Click on the scatter plot to see the forward pass with actual values, or hit Auto to watch training evolve the decision boundary in real time.

What is a neuron?

A neural network starts with one simple idea: take some inputs, multiply each by a weight, add a bias, and pass the result through a function. That is a neuron.

A single neuron computes two things. First, the weighted sum:

$z = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n + b = \mathbf{w}^\top \mathbf{x} + b$

Then, the activation:

$a = \sigma(z)$

Here $\mathbf{x}$ is the input vector, $\mathbf{w}$ is the weight vector, $b$ is the bias, and $\sigma$ is the activation function.

The dot product $\mathbf{w}^\top \mathbf{x}$ measures how much the input aligns with what the neuron is looking for. The bias shifts the decision boundary. The activation function introduces nonlinearity.

graph LR
  x1((x₁)) -->|w₁| S["∑ + b"]
  x2((x₂)) -->|w₂| S
  x3((x₃)) -->|w₃| S
  S -->|z| A["σ(z)"]
  A -->|a| O((output))

If you have seen logistic regression, you already know what a neuron does. Logistic regression computes $\sigma(\mathbf{w}^\top \mathbf{x} + b)$ where $\sigma$ is the sigmoid function. A neuron is the same computation, except you can swap sigmoid for other activation functions.

Why activation functions matter

Here is the key insight: without activation functions, a deep network is just a single linear transformation. It does not matter how many layers you stack.

Suppose you have two linear layers:

$\mathbf{h} = W_1 \mathbf{x} + \mathbf{b}_1$

$\mathbf{y} = W_2 \mathbf{h} + \mathbf{b}_2$

Substitute the first equation into the second:

$\mathbf{y} = W_2 (W_1 \mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2 = (W_2 W_1) \mathbf{x} + (W_2 \mathbf{b}_1 + \mathbf{b}_2)$

This is just $\mathbf{y} = W' \mathbf{x} + \mathbf{b}'$ where $W' = W_2 W_1$. No matter how many linear layers you stack, the result collapses to a single linear layer. We prove this concretely with numbers in Example 3 below.

Activation functions break this linearity. They let the network learn curved decision boundaries, complex patterns, and subtle relationships in data. Every hidden layer needs a nonlinear activation, or you are wasting depth.

Common activation functions

Activation functions compared: ReLU, sigmoid, and tanh over the range x = -5 to 5.

Sigmoid

$\sigma(z) = \frac{1}{1 + e^{-z}}$

Squashes any real number into the range $(0, 1)$. Historically popular because it resembles a probability. The problem: for large $|z|$, the gradient approaches zero. This makes learning very slow in deep networks. The phenomenon is called the vanishing gradient problem.

Tanh

$\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}$

Output range is $(-1, 1)$. Zero-centered outputs help gradient descent converge faster than sigmoid. Still suffers from vanishing gradients at the extremes, though.

ReLU (Rectified Linear Unit)

$\text{ReLU}(z) = \max(0, z)$

Dead simple. If $z$ is positive, pass it through. If negative, output zero. ReLU solved a huge practical problem: gradients do not vanish for positive inputs because the gradient is exactly 1. Training became much faster. The downside: neurons can “die” if they consistently receive negative inputs, because the gradient is exactly 0 there.

Leaky ReLU

$\text{LeakyReLU}(z) = \begin{cases} z & \text{if } z > 0 \\ \alpha z & \text{if } z \leq 0 \end{cases}$

where $\alpha$ is a small constant like 0.01. This fixes the dying ReLU problem by allowing a small gradient when $z < 0$.

Activation function comparison

For most problems, start with ReLU in hidden layers. Use sigmoid or softmax in the output layer for classification. That is a solid default.

How layers compose

A single neuron can only learn a linear boundary (with a nonlinear squash). To learn complex patterns, we stack neurons into layers, and layers into networks.

A feedforward neural network has three types of layers:

1. Input layer: holds your raw features. No computation happens here.
2. Hidden layers: each neuron takes all outputs from the previous layer, computes a weighted sum plus bias, and applies an activation function.
3. Output layer: produces the final prediction. For regression, typically no activation or a linear one. For binary classification, sigmoid. For multi-class, softmax.

graph LR
  subgraph Input
      i1((x₁))
      i2((x₂))
  end
  subgraph Hidden["Hidden layer (3 units)"]
      h1((h₁))
      h2((h₂))
      h3((h₃))
  end
  subgraph Output
      o1((ŷ))
  end
  i1 --> h1
  i1 --> h2
  i1 --> h3
  i2 --> h1
  i2 --> h2
  i2 --> h3
  h1 --> o1
  h2 --> o1
  h3 --> o1

In matrix notation, a hidden layer computes:

$\mathbf{h} = \sigma(W \mathbf{x} + \mathbf{b})$

where $W$ is the weight matrix, $\mathbf{b}$ is the bias vector, and $\sigma$ is applied element-wise. Each row of $W$ contains the weights for one neuron.

The full network is function composition:

$\hat{y} = f_L(f_{L-1}(\cdots f_2(f_1(\mathbf{x}))))$

Each $f_l$ represents one layer’s operation: linear transform followed by an activation.

Universal approximation theorem

Here is a remarkable result: a feedforward network with a single hidden layer containing enough neurons can approximate any continuous function on a compact domain to arbitrary accuracy.

What this means in plain terms: neural networks are theoretically powerful enough to learn any reasonable input-output mapping. Given enough hidden neurons, the network can get as close as you want to the true function.

What it does not mean:

• It does not tell you how many neurons you need. The number could be astronomically large.
• It does not guarantee that gradient descent will find the right weights.
• It says nothing about generalization to unseen data.

In practice, we use deeper networks (more layers) rather than extremely wide ones (many neurons in a single layer). Depth lets networks build hierarchical representations. Early layers learn simple patterns. Later layers combine them into complex features.

Example 1: Single neuron forward pass

Given: $\mathbf{x} = [2, 3, -1]$, $\mathbf{w} = [0.5, -0.2, 0.8]$, $b = 0.1$. Compute the output using ReLU.

Step 1: Weighted sum plus bias.

$z = w_1 x_1 + w_2 x_2 + w_3 x_3 + b$

$z = (0.5)(2) + (-0.2)(3) + (0.8)(-1) + 0.1$

$z = 1.0 - 0.6 - 0.8 + 0.1 = -0.3$

Step 2: Apply ReLU.

$a = \text{ReLU}(-0.3) = \max(0, -0.3) = 0$

The neuron outputs 0. Because $z$ is negative, ReLU kills the signal completely. This neuron, with these particular weights, does not “fire” for this input.

Example 2: Forward pass through a 2-layer network

Network: 2 inputs, 3 hidden units with ReLU, 1 output with linear activation.

Given:

Input: $\mathbf{x} = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$

Hidden layer weights and biases:

$W^{(1)} = \begin{bmatrix} 0.2 & -0.1 \\ 0.4 & 0.3 \\ -0.5 & 0.6 \end{bmatrix}, \quad \mathbf{b}^{(1)} = \begin{bmatrix} 0.1 \\ -0.2 \\ 0.0 \end{bmatrix}$

Output layer weights and bias:

$\mathbf{w}^{(2)} = \begin{bmatrix} 0.7 \\ -0.3 \\ 0.5 \end{bmatrix}, \quad b^{(2)} = 0.1$

Step 1: Compute hidden layer pre-activations.

$\mathbf{z}^{(1)} = W^{(1)} \mathbf{x} + \mathbf{b}^{(1)}$

$z_1^{(1)} = (0.2)(1) + (-0.1)(2) + 0.1 = 0.2 - 0.2 + 0.1 = 0.1$

$z_2^{(1)} = (0.4)(1) + (0.3)(2) + (-0.2) = 0.4 + 0.6 - 0.2 = 0.8$

$z_3^{(1)} = (-0.5)(1) + (0.6)(2) + 0.0 = -0.5 + 1.2 = 0.7$

Step 2: Apply ReLU.

$\mathbf{h} = \text{ReLU}(\mathbf{z}^{(1)}) = \begin{bmatrix} \max(0, 0.1) \\ \max(0, 0.8) \\ \max(0, 0.7) \end{bmatrix} = \begin{bmatrix} 0.1 \\ 0.8 \\ 0.7 \end{bmatrix}$

All values are positive, so all three neurons are active.

Step 3: Compute output.

$\hat{y} = (\mathbf{w}^{(2)})^\top \mathbf{h} + b^{(2)}$

$\hat{y} = (0.7)(0.1) + (-0.3)(0.8) + (0.5)(0.7) + 0.1$

$\hat{y} = 0.07 - 0.24 + 0.35 + 0.1 = 0.28$

The network outputs 0.28. Every hidden neuron contributed to the final answer, weighted by the output layer.

Example 3: Why stacking linear layers collapses

Let us prove that stacking linear layers without activations gives you nothing extra. Take two layers with $2 \times 2$ weight matrices and no biases for clarity.

Given:

$W_1 = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}, \quad W_2 = \begin{bmatrix} 0.5 & -1 \\ 1.5 & 0.5 \end{bmatrix}$

Input: $\mathbf{x} = \begin{bmatrix} 1 \\ 1 \end{bmatrix}$

Two separate layers:

$\mathbf{h} = W_1 \mathbf{x} = \begin{bmatrix} 1 + 2 \\ 3 + 4 \end{bmatrix} = \begin{bmatrix} 3 \\ 7 \end{bmatrix}$

$\mathbf{y} = W_2 \mathbf{h} = \begin{bmatrix} (0.5)(3) + (-1)(7) \\ (1.5)(3) + (0.5)(7) \end{bmatrix} = \begin{bmatrix} 1.5 - 7 \\ 4.5 + 3.5 \end{bmatrix} = \begin{bmatrix} -5.5 \\ 8.0 \end{bmatrix}$

Single combined layer:

$W' = W_2 W_1 = \begin{bmatrix} (0.5)(1)+(-1)(3) & (0.5)(2)+(-1)(4) \\ (1.5)(1)+(0.5)(3) & (1.5)(2)+(0.5)(4) \end{bmatrix} = \begin{bmatrix} -2.5 & -3.0 \\ 3.0 & 5.0 \end{bmatrix}$

$\mathbf{y} = W' \mathbf{x} = \begin{bmatrix} -2.5 - 3.0 \\ 3.0 + 5.0 \end{bmatrix} = \begin{bmatrix} -5.5 \\ 8.0 \end{bmatrix}$

Same result. Two linear layers without activation are mathematically identical to one linear layer with $W' = W_2 W_1$. Add 100 layers, still the same story. This is exactly why activation functions are non-negotiable.

What comes next

You now know what a neuron computes and how layers stack to form networks. But how does a network actually learn? How do we adjust the weights so the output matches what we want?

That is backpropagation: the algorithm that computes gradients through the network so gradient descent can update every weight. It is the engine behind all of deep learning.