wunder beta

📘 How does a neural network actually learn?

Neurons, activations, and backprop—how a network turns error into an update, not a metaphor.

4
lessons
~20 min
to learn
Adults
level
Start the course →

What you’ll learn

  1. The Artificial Neuron and ActivationsExplain how a single artificial neuron computes its output and why nonlinear activation functions are essential.This lesson introduces the artificial neuron as a weighted sum of inputs plus a bias, followed by a nonlinear activation. It shows why nonlinearity matters: stacking purely linear layers collapses to a single linear function. It compares the sigmoid and tanh squashing functions with ReLU. It connects the saturating slopes of sigmoid and tanh to the vanishing-gradient problem and explains why ReLU, defined as max of zero and the input, became the modern default.
  2. The Multilayer PerceptronDescribe how neurons combine into a feedforward multilayer perceptron and what the universal approximation idea does and does not promise.This lesson assembles neurons into input, hidden, and output layers forming a feedforward multilayer perceptron, and traces the forward pass that turns an input into a prediction. It presents the universal approximation theorem conceptually: a wide enough single-hidden-layer network can approximate any continuous function on a bounded region, without guaranteeing trainability or efficiency. It introduces output layers and the loss functions that score them, namely mean squared error for regression and cross-entropy for classification.
  3. How Networks Learn: Gradient Descent and BackpropExplain conceptually how gradient descent and backpropagation work together to train a network.This lesson frames learning as minimizing a loss surface over the network's parameters. It explains gradient descent, the role of the learning rate, and the consequences of setting it too high or too low. It describes backpropagation as the chain-rule-based algorithm that computes each weight's gradient by propagating error backward after the forward pass. It distinguishes computing gradients (backprop) from using them to update weights (descent), and introduces mini-batch stochastic gradient descent and epochs.
  4. Generalization and the Socratic CheckDiagnose underfitting and overfitting and reason about which regularization techniques address them.This lesson defines underfitting and overfitting and shows how to recognize each from training and validation curves. It surveys regularization techniques, including weight decay (L2), dropout, and early stopping, and explains the bias-variance tradeoff they manage. The final segment is a Socratic, applied check that asks learners to reason through scenarios, connecting neurons, activations, loss, backpropagation, and regularization into a coherent picture of how trained networks generalize.

Questions this course answers

What does an artificial neuron compute before applying its activation function?

A neuron forms a weighted sum of its inputs and adds a bias to produce the pre-activation value, which is then passed through the activation function.

Why would a deep network made of only linear layers, with no nonlinear activations, gain nothing from extra depth?

Composing linear maps yields another linear map, so any number of purely linear layers is equivalent to one linear layer; nonlinearity is what makes depth useful.

What is the main reason ReLU helps avoid the vanishing-gradient problem compared to sigmoid?

For positive inputs ReLU's derivative is one, so gradients pass through without the repeated shrinking caused by the near-zero slopes of saturated sigmoid or tanh units.

In a feedforward multilayer perceptron, how does information flow during prediction?

A feedforward network passes information in a single direction from the input layer through the hidden layers to the output layer; there are no cycles in the forward pass.

What does the universal approximation theorem (Cybenko 1989, Hornik 1991) actually guarantee?

The theorem guarantees expressive capacity: a sufficiently wide single-hidden-layer network can approximate any continuous function on a closed bounded region. It says nothing about how many neurons are needed or whether training will find the weights.

Which loss function is the standard choice for a classification task with probability outputs?

Cross-entropy measures the distance between predicted class probabilities and the true label and is the standard classification loss; mean squared error is typically used for regression.

Grounded in trusted sources

  • Goodfellow, Bengio & Courville, Deep Learning (MIT Press, 2016), ch. 6
  • Nielsen, Neural Networks and Deep Learning (2015), ch. 1
  • Cybenko, Approximation by Superpositions of a Sigmoidal Function, Math. Control Signals Systems (1989)
  • LeCun, Bengio & Hinton, Deep learning, Nature 521 (2015)
  • Goodfellow, Bengio & Courville, Deep Learning (MIT Press, 2016), ch. 6 & 8
  • Nielsen, Neural Networks and Deep Learning (2015), ch. 2

Every Wunder lesson is built from real, reputable sources — never invented.

Related courses

📘How does a neural net learn?📘How do machines read language?📘Keep transformer from meaning RNN clone📘Keep fine-tuning from meaning scratch📘Keep prompt from meaning magic spell📘How do you evaluate an AI product in the wild?📘How do humans stay in an AI loop?📘How do you manage risk in an AI product?

Wunder is a personalized learn-anything platform — tell it any topic and it builds a beautiful, fact-checked course in minutes, with narration, a knowledge check, and a college-style University track.

All topics · Home

© 2026 Wunder Learning LLC · Terms & Privacy