📘 How does a neural net learn?

What you’ll learn

1. From Neuron to PerceptronExplain how a single artificial neuron computes a weighted sum plus bias, applies an activation function, and why a linear unit alone is fundamentally limited.An artificial neuron is a simple computational unit: it forms a weighted sum of its inputs, adds a bias, and passes the result through an activation function. The classic perceptron learns linear decision boundaries, which is why it cannot solve non-linearly-separable problems such as XOR. This limitation, articulated by Minsky and Papert, motivates both non-linear activations and the multi-layer architectures explored in later lessons.
2. Stacking Layers: The Multi-Layer PerceptronDescribe how neurons organize into input, hidden, and output layers to perform a forward pass, and articulate why depth and width expand representational power.A multi-layer perceptron arranges neurons into successive layers, where each layer transforms the previous layer's activations via a matrix multiply, bias add, and non-linear activation. The forward pass propagates an input through these layers to an output, and hidden layers let the network compose simple features into complex ones. The universal approximation theorem guarantees that even one sufficiently wide hidden layer can approximate any continuous function, though depth often achieves the same expressiveness far more efficiently.
3. Loss, Gradients, and BackpropagationExplain how a loss function quantifies error and how backpropagation applies the chain rule to compute gradients of that loss with respect to every weight.Training reduces to minimizing a loss function that measures the gap between predictions and targets, such as mean squared error for regression or cross-entropy for classification. Backpropagation, formalized for neural networks by Rumelhart, Hinton, and Williams in 1986, efficiently computes the gradient of the loss with respect to every parameter by applying the chain rule backward through the network. These gradients tell each weight which direction and how strongly to change to reduce error, forming the basis for the optimization studied next.
4. Training with Gradient DescentDescribe how gradient descent and its stochastic and mini-batch variants update weights, and explain the role of learning rate, epochs, and modern optimizers like Adam.Gradient descent iteratively nudges weights opposite their gradient, scaled by a learning rate, to reduce loss. Stochastic and mini-batch variants estimate the gradient from small samples, trading some accuracy for far greater speed and the ability to scale to large datasets. The learning rate is the single most consequential hyperparameter, and adaptive optimizers such as Adam (Kingma and Ba, 2015) combine momentum and per-parameter scaling to make training more robust.
5. Generalization, Overfitting, and RegularizationDistinguish memorization from generalization and explain how data splits, regularization, dropout, and early stopping help a network perform well on unseen data.A network's real goal is generalization: low error on data it has never seen, not just on the training set. Overfitting occurs when a model memorizes training noise, which is detected by a growing gap between training and validation performance. Techniques including held-out validation and test splits, L2 weight regularization, dropout, early stopping, and data augmentation constrain the model so it captures genuine structure rather than idiosyncrasies, trading a controlled amount of bias for reduced variance.
6. Capstone: Build and Critique a Neural Network ClassifierGuide the learner to build, train, and document a small neural-network classifier as a mini artifact, then prepare it for structured peer critique.In this capstone you assemble the full pipeline from earlier lessons into a working classifier on a small, well-understood dataset, then document your design choices and results for peer review. The deliverable is a reproducible mini artifact: a trained model, its training and validation curves, and a short rationale connecting your architecture, loss, optimizer, and regularization decisions to the concepts taught. You will then both give and receive critique using a shared rubric, the core assessment of this course.

Questions this course answers

Grounded in trusted sources

• Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning (MIT Press) — MLP, loss, backprop
• Michael Nielsen, Neural Networks and Deep Learning (online book) — perceptron to backprop intuition, http://neuralnetworksanddeeplearning.com/
• David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams, “Learning representations by back-propagating errors,” Nature (1986)
• 3Blue1Brown, Neural networks series — visual gradient descent intuition, https://www.3blue1brown.com/
• Stanford CS231n course notes — loss functions and optimization overview, https://cs231n.github.io/

Every Wunder lesson is built from real, reputable sources — never invented.

Related courses