Neural Networks

"Deep learning" may sound mysterious and complex, but its basic building block—the neuron—is simply a weighted sum followed by a nonlinear transformation. Starting with the perceptron, this chapter guides you step by step through multilayer networks, activation functions, forward propagation, and backpropagation, and shows you how to implement an MLP from scratch using pure Python that can solve the XOR problem.

1. What You'll Learn

• Perceptron: The simplest neural network unit
• Structure and Intuition of Multi-Layer Perceptrons (MLPs)
• The Role and Selection of Activation Functions (Sigmoid / ReLU / Tanh)
• The Core Intuition Behind Forward Propagation and Backpropagation
• Deep learning = a stack of multilayer neural networks

2. Story: The "Brain" Behind 10 Lines of Code

(1) Pain Point: What Exactly Is Deep Learning?

Bob had heard that "deep learning" could recognize faces, translate languages, and write articles, but he had absolutely no idea what a neural network was. He tried looking it up on Wikipedia, but the screen full of partial derivatives and the chain rule made him close the page right away. "Can't someone just explain to me in the simplest terms what a 'neuron' actually does?"

(2) Alice's 10 lines of code

Alice smiled as she opened Python and wrote a perceptron in 10 lines of code—it takes two numbers as input and outputs either 0 or 1:

# A single perceptron: AND gate
def perceptron(x1, x2, w1=1, w2=1, b=-1.5):
    return 1 if w1 * x1 + w2 * x2 + b > 0 else 0

for x1, x2 in [(0,0),(0,1),(1,0),(1,1)]:
    print(f"AND({x1},{x2}) = {perceptron(x1, x2)}")

AND(0,0) = 0
AND(0,1) = 0
AND(1,0) = 0
AND(1,1) = 1

Bob realized, “Isn’t this just weighted voting? Both inputs are set to 1, and if the weighted sum exceeds the threshold, the output is 1—so a neural network is just a series of these simple decision units connected together!”

(3) Benefits: From Simple Units to Powerful Networks

Alice nodded. "That's right. A single perceptron can only perform linear classification, but by stacking dozens or hundreds of perceptrons into a multi-layer network, we can fit extremely complex functions—and that's where deep learning begins."

3. Perceptron—The Simplest Neural Network

(1) Mathematical Definition of a Perceptron

The perceptron, proposed by psychologist Rosenblatt in 1958, is one of the earliest models of an artificial neuron. It receives multiple inputs $x_1, x_2, \ldots, x_n$, each with a weight $w_i$, plus a bias $b$, and finally outputs 0 or 1 via a step function:

$$y = \text{step}\left(\sum_{i=1}^{n} w_i x_i + b\right)$$

In particular, $\text{step}(z) = 1$ if $z > 0$, and $0$ otherwise.

Intuition: Imagine you're deciding whether to go for a run—the temperature, air quality, whether you have time, and your mood—each factor has a "weight," and if the total exceeds your "laziness threshold," you head out.

(2) The Geometric Meaning of the Perceptron

A perceptron draws a straight line $w_1 x_1 + w_2 x_2 + b = 0$ in two-dimensional space, outputting 1 on one side of the line and 0 on the other. It can perfectly solve linearly separable problems such as AND and OR, but it cannot solve the XOR problem—because the two classes of points in an XOR problem cannot be separated by a single straight line.

▶ Example: A Pure Python Implementation of a Perceptron AND Gate (Difficulty: ⭐)

# Perceptron AND gate with training
def step(z):
    return 1 if z > 0 else 0

def train_perceptron(X, y, lr=0.1, epochs=10):
    n_features = len(X[0])
    weights = [0.0] * n_features
    bias = 0.0
    for epoch in range(epochs):
        total_error = 0
        for xi, yi in zip(X, y):
            z = sum(w * x for w, x in zip(weights, xi)) + bias
            pred = step(z)
            error = yi - pred
            total_error += abs(error)
            for j in range(n_features):
                weights[j] += lr * error * xi[j]
            bias += lr * error
        if total_error == 0:
            break
    return weights, bias

X = [[0,0],[0,1],[1,0],[1,1]]
y_and = [0, 0, 0, 1]
w, b = train_perceptron(X, y_and)
print(f"Weights: {w}, Bias: {b}")
for xi in X:
    print(f"AND{xi} = {step(sum(ww*xx for ww,xx in zip(w,xi)) + b)}")

Weights: [0.2, 0.1], Bias: -0.2
AND[0, 0] = 0
AND[0, 1] = 0
AND[1, 0] = 0
AND[1, 1] = 1

4. Multi-Layer Perceptron (MLP) Architecture

(1) Why Are Multiple Layers Needed?

A single-layer perceptron can only draw straight lines and cannot solve the XOR problem. However, if the outputs of two perceptrons are fed into a third perceptron, it becomes possible to combine two straight lines to form a nonlinear boundary. This is the core idea behind the multi-layer perceptron (MLP).

(2) The Three-Layer Structure of an MLP

Every node in each layer is connected to all nodes in the previous layer (fully connected), and the strength of these connections is represented by weights.

(3) The Forward Propagation Process in Neural Networks

graph LR
    X1["x₁"] --> H1["h₁ (sigmoid)"]
    X1 --> H2["h₂ (sigmoid)"]
    X2["x₂"] --> H1
    X2 --> H2
    H1 --> Y["y (output)"]
    H2 --> Y
    style X1 fill:#e1f5fe
    style X2 fill:#e1f5fe
    style H1 fill:#fff9c4
    style H2 fill:#fff9c4
    style Y fill:#c8e6c9

graph LR
    X1["x₁"] --> H1["h₁ (sigmoid)"]
    X1 --> H2["h₂ (sigmoid)"]
    X2["x₂"] --> H1
    X2 --> H2
    H1 --> Y["y (output)"]
    H2 --> Y
    style X1 fill:#e1f5fe
    style X2 fill:#e1f5fe
    style H1 fill:#fff9c4
    style H2 fill:#fff9c4
    style Y fill:#c8e6c9

Intuition: The input layer is like the eyes, the hidden layers are like the brain’s processing regions, and the output layer is like the mouth speaking the answer. Each layer performs a "weighted sum + nonlinear transformation."

(4) Comparison: Perceptron vs. MLP vs. Deep Neural Networks

5. Activation Functions—Adding Nonlinearity to the Network

(1) Why Are Activation Functions Needed?

Without activation functions, regardless of the number of layers, a network is essentially a linear transformation—the composition of multiple linear transformations remains linear. Activation functions introduce nonlinearity, enabling the network to fit complex curves and boundaries.

Analogy: Linear transformations are like "stretching and rotating"; no matter how many times you stretch or rotate them, the result is still a straight line or a plane. Activation functions are like "folding"—fold a piece of paper once, and the space changes.

(2) Comparison of Common Activation Functions

▶ Example: Comparing the Output Values of Different Activation Functions (Difficulty: ⭐)

# Compare activation functions
import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def tanh(z):
    return np.tanh(z)

def relu(z):
    return np.maximum(0, z)

def leaky_relu(z, alpha=0.01):
    return np.where(z > 0, z, alpha * z)

z_values = np.array([-5, -2, -1, 0, 1, 2, 5])
print(f"{'z':>5} | {'Sigmoid':>8} | {'Tanh':>8} | {'ReLU':>8} | {'LeakyReLU':>10}")
print("-" * 55)
for z in z_values:
    s = sigmoid(z)
    t = tanh(z)
    r = float(relu(np.array([z]))[0])
    lr = float(leaky_relu(np.array([z]))[0])
    print(f"{z:5.1f} | {s:8.4f} | {t:8.4f} | {r:8.4f} | {lr:10.4f}")

    z |  Sigmoid |     Tanh |     ReLU |  LeakyReLU
-------------------------------------------------------
 -5.0 |   0.0067 |  -0.9999 |   0.0000 |    -0.0500
 -2.0 |   0.1192 |  -0.9640 |   0.0000 |    -0.0200
 -1.0 |   0.2689 |  -0.7616 |   0.0000 |    -0.0100
  0.0 |   0.5000 |   0.0000 |   0.0000 |     0.0000
  1.0 |   0.7311 |   0.7616 |   1.0000 |     1.0000
  2.0 |   0.8808 |   0.9640 |   2.0000 |     2.0000
  5.0 |   0.9933 |   0.9999 |   5.0000 |     5.0000

6. Forward Propagation and Backpropagation

(1) Forward propagation: From input to output

Forward propagation is the process by which data travels from the input layer through each layer to the output layer:

1. Calculate the hidden layer input: $z^{(1)} = W^{(1)} \cdot x + b^{(1)}$
2. Apply the activation function: $h = \sigma(z^{(1)})$
3. Calculate the output layer input: $z^{(2)} = W^{(2)} \cdot h + b^{(2)}$
4. Output prediction: $\hat{y} = \sigma(z^{(2)})$

(2) Loss Function: Measures the discrepancy between the prediction and the true value

Mean Square Error (MSE) is commonly used as a loss function:

Loss is the square of the difference between the actual value and the predicted value.

The smaller the loss, the more accurate the prediction. The goal of training is to find the weights that minimize the loss.

(3) Backpropagation: From Error to Weight Updates

The core idea of backpropagation: Starting from the error in the output layer, calculate the "contribution" of each weight to the error by propagating backward through the network layer by layer, and then adjust the weights.

Intuitive Analogy: A factory has a problem with its products—

• First, check how many errors occurred in the final step (the output layer)
• Ask again how many errors were caused by the previous process (hidden layer)
• Trace back layer by layer until you reach the raw materials (input layer)
• Each process should adjust its operating procedures based on the errors it has contributed to

Mathematically, this is the Chain Rule: $\frac{\partial L}{\partial w} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z} \cdot \frac{\partial z}{\partial w}$

(4) Table of Key Concepts in Backpropagation

(5) Intuition for Loss Surfaces

Imagine you're standing on a mountain, and your goal is to reach the bottom of the valley (minimum loss). With every step, you head down the steepest slope—that's gradient descent. The learning rate determines the size of your steps:

• Stride too long: You might end up crossing the valley floor and landing on the opposite hillside
• Stride too short: You've been walking for ages and you're still only halfway up the mountain.
• Optimal stride: Reaching the valley floor steadily and quickly

7. Quickly Try Out an MLP with sklearn

▶ Example: Classification using sklearn's MLPClassifier (Difficulty: ⭐)

# MLPClassifier on XOR problem
from sklearn.neural_network import MLPClassifier
import numpy as np

X = np.array([[0,0],[0,1],[1,0],[1,1]])
y = np.array([0, 1, 1, 0])

mlp = MLPClassifier(hidden_layer_sizes=(4,), activation='relu',
                    max_iter=2000, random_state=42)
mlp.fit(X, y)

print("XOR Predictions:")
for xi in X:
    print(f"  {xi} -> {mlp.predict([xi])[0]}")
print(f"Accuracy: {mlp.score(X, y):.2f}")

XOR Predictions:
  [0 0] -> 0
  [0 1] -> 1
  [1 0] -> 1
  [1 1] -> 0
Accuracy: 1.00

▶ Example: Visualizing Changes in Decision Boundaries (Difficulty: ⭐⭐)

# Visualize MLP decision boundary on XOR
import numpy as np
import matplotlib.pyplot as plt
from sklearn.neural_network import MLPClassifier

X = np.array([[0,0],[0,1],[1,0],[1,1]])
y = np.array([0, 1, 1, 0])

fig, axes = plt.subplots(1, 3, figsize=(15, 4))
hidden_configs = [(2,), (4,), (8,)]

for ax, config in zip(axes, hidden_configs):
    mlp = MLPClassifier(hidden_layer_sizes=config, activation='relu',
                        max_iter=3000, random_state=42)
    mlp.fit(X, y)

    xx, yy = np.meshgrid(np.linspace(-0.5, 1.5, 100),
                         np.linspace(-0.5, 1.5, 100))
    Z = mlp.predict(np.c_[xx.ravel(), yy.ravel()]).reshape(xx.shape)

    ax.contourf(xx, yy, Z, alpha=0.3, cmap='coolwarm')
    ax.scatter(X[:,0], X[:,1], c=y, cmap='coolwarm', edgecolors='k', s=100)
    ax.set_title(f"Hidden: {config}, Acc: {mlp.score(X,y):.2f}")
    ax.set_xlabel("x1")
    ax.set_ylabel("x2")

plt.tight_layout()
plt.savefig("mlp_decision_boundary.png", dpi=100)
plt.show()

▶ Example: How the Number of Nodes in a Hidden Layer Affects Performance (Difficulty: ⭐⭐)

# Effect of hidden layer size on training loss
import numpy as np
from sklearn.neural_network import MLPClassifier
import matplotlib.pyplot as plt

X = np.array([[0,0],[0,1],[1,0],[1,1]])
y = np.array([0, 1, 1, 0])

fig, ax = plt.subplots(figsize=(8, 5))
for n_hidden in [2, 4, 8, 16]:
    mlp = MLPClassifier(hidden_layer_sizes=(n_hidden,), activation='relu',
                        max_iter=3000, random_state=42, solver='lbfgs')
    mlp.fit(X, y)
    ax.plot(range(len(mlp.loss_curve_)), mlp.loss_curve_,
            label=f"Hidden={n_hidden}")

ax.set_xlabel("Iteration")
ax.set_ylabel("Loss")
ax.set_title("Training Loss vs Hidden Layer Size")
ax.legend()
ax.set_yscale('log')
plt.savefig("hidden_size_loss.png", dpi=100)
plt.show()

8. Deep Learning = Multi-Layer Neural Networks

(1) Comparison of Traditional ML and Deep Learning

(2) What does "deep" mean?

"Depth" refers to the number of hidden layers. It is generally agreed that:

• Shallow network: 1 hidden layer (MLP)
• Deep Neural Networks: 2 or more hidden layers
• Ultra-deep networks: dozens or even hundreds of layers (ResNet-152 has 152 layers)

The more layers there are, the more abstract the features the network can extract—the first layer might learn edges, the second layer might learn textures, and the third layer might learn components; as we move further down the network, the features become increasingly "semantic."

(3) More depth isn't necessarily better

A deeper network does not necessarily mean better performance. Too many layers can lead to:

• Gradient vanishing/explosion: The signal attenuates or amplifies as it is passed through successive layers
• Overfitting: Too many parameters, causing the model to memorize the training data rather than learn the underlying patterns
• Training Cost: More parameters = longer training time + more computing power

The model depth should be chosen based on the task’s complexity and the amount of data—just like building a skyscraper: taller isn’t always better; you have to consider the foundation and the budget.

9. Comprehensive Hands-On Exercise: Implementing an MLP Classifier from Scratch

▶ Example: Implementing an MLP from Scratch to Solve the XOR Problem (Difficulty: ⭐⭐⭐)

Below, we will implement a complete MLP using pure Python (without any frameworks), including forward propagation, backpropagation, and the training loop, with the goal of solving the XOR problem.

# MLP from scratch: solve XOR with pure Python + NumPy
import numpy as np

# --- Activation functions and their derivatives ---
def sigmoid(z):
    return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

def sigmoid_deriv(a):
    return a * (1 - a)

# --- MLP class ---
class MLP:
    def __init__(self, layer_sizes):
        # layer_sizes e.g. [2, 4, 1] => 2-input, 4-hidden, 1-output
        self.weights = []
        self.biases = []
        for i in range(len(layer_sizes) - 1):
            w = np.random.randn(layer_sizes[i], layer_sizes[i+1]) * 0.5
            b = np.zeros((1, layer_sizes[i+1]))
            self.weights.append(w)
            self.biases.append(b)

    def forward(self, X):
        self.activations = [X]
        for i in range(len(self.weights)):
            z = self.activations[-1] @ self.weights[i] + self.biases[i]
            a = sigmoid(z)
            self.activations.append(a)
        return self.activations[-1]

    def backward(self, y, lr=0.5):
        m = y.shape[0]
        deltas = [None] * len(self.weights)

        # Output layer delta
        output = self.activations[-1]
        deltas[-1] = (output - y) * sigmoid_deriv(output)

        # Hidden layer deltas (backprop)
        for i in range(len(self.weights) - 2, -1, -1):
            deltas[i] = (deltas[i+1] @ self.weights[i+1].T) * sigmoid_deriv(self.activations[i+1])

        # Update weights and biases
        for i in range(len(self.weights)):
            self.weights[i] -= lr * (self.activations[i].T @ deltas[i]) / m
            self.biases[i] -= lr * np.mean(deltas[i], axis=0, keepdims=True)

    def train(self, X, y, epochs=5000, lr=0.5):
        self.losses = []
        for epoch in range(epochs):
            pred = self.forward(X)
            loss = np.mean((y - pred) ** 2) / 2
            self.losses.append(loss)
            self.backward(y, lr)
            if epoch % 1000 == 0:
                print(f"Epoch {epoch:4d} | Loss: {loss:.6f}")

    def predict(self, X):
        return (self.forward(X) > 0.5).astype(int)

# --- Train on XOR ---
np.random.seed(42)
X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=float)
y = np.array([[0],[1],[1],[0]], dtype=float)

mlp = MLP([2, 4, 1])
mlp.train(X, y, epochs=5000, lr=1.0)

print("\nFinal Predictions:")
preds = mlp.predict(X)
for i in range(4):
    print(f"  XOR{X[i].astype(int).tolist()} = {preds[i,0]} (target: {y[i,0].astype(int)})")

Epoch    0 | Loss: 0.132415
Epoch 1000 | Loss: 0.004726
Epoch 2000 | Loss: 0.002517
Epoch 3000 | Loss: 0.001751
Epoch 4000 | Loss: 0.001330

Final Predictions:
  XOR[0, 0] = 0 (target: 0)
  XOR[0, 1] = 1 (target: 1)
  XOR[1, 0] = 1 (target: 1)
  XOR[1, 1] = 0 (target: 0)

❓ FAQ

📖 Summary

• Perceptron: Weighted sum + step function, the simplest linear classifier
• MLP: Multi-Layer Perceptron; hidden layers provide nonlinear capabilities
• Activation Function: Introduces nonlinearity; ReLU is the modern default choice
• Forward Propagation: Data is processed layer by layer from the input layer to the output layer
• Backpropagation: The error is propagated backward layer by layer from the output layer to update the weights
• Deep Learning: Multilayer Neural Networks: Automatically Learning Multi-Level Features
• Gradient Descent: Descend along the steepest slope of the loss surface to find the optimal weights

Key Concept: A neuron = weighted sum + nonlinear transformation. A neural network = many neurons stacked together. Deep learning = a neural network with many layers. From a perceptron with 10 lines of code to GPT, they all essentially follow the same paradigm.

📝 Exercises

1. Basics (⭐): Implement a perceptron for an OR gate. The truth table for the OR gate is: (0,0) → 0, (0,1) → 1, (1,0) → 1, (1,1) → 1. Modify the labels in the perceptron training code from this chapter, then train the model and verify the results.

# Starter code: OR gate perceptron
X = [[0,0],[0,1],[1,0],[1,1]]
y_or = [0, 1, 1, 1]
# TODO: train and test

2. Advanced (⭐⭐): Use sklearn's MLPClassifier to perform classification on the Iris dataset, tune the parameters, and record the accuracy for different configurations:

# Starter code: MLP on Iris
from sklearn.datasets import load_iris
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import train_test_split

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.3, random_state=42)

# TODO: try different hidden_layer_sizes, activation, learning_rate
# Record accuracy for each config

Try at least 3 different hidden_layer_sizes configurations (such as (10,), (50,50), (100,)) and 2 activation functions, and note which configuration works best.

3. Challenge (⭐⭐⭐): Based on the pure Python MLP implementation in this chapter, plot the loss curve for the manually implemented MLP:

# After training MLP from Section 9, plot the loss curve
import matplotlib.pyplot as plt

# mlp.losses is already recorded during training
plt.figure(figsize=(8, 5))
plt.plot(range(len(mlp.losses)), mlp.losses, linewidth=1)
plt.xlabel("Epoch")
plt.ylabel("Loss (MSE)")
plt.title("MLP Training Loss Curve (XOR)")
plt.yscale('log')
plt.grid(True, alpha=0.3)
plt.savefig("mlp_loss_curve.png", dpi=100)
plt.show()

Advanced Extensions: ① Compare the loss curves for different learning rates (0.1, 0.5, 1.0, 2.0) on the same graph; ② Experiment with different hidden layer sizes (2, 4, 8) to see how they affect the convergence rate; ③ Observe whether the loss oscillates or even diverges when the learning rate is too high.

Web-Tutorial Tech Team

A team of developers maintaining programming tutorials. Each tutorial is written and reviewed by developers with expertise in that field. We work to keep our content accurate and reliable — if you spot an issue, please let us know.

About Us