🧠 Deep Learning

What you’ll learn

1. What 'Deep' Actually MeansFrame the through-line: a deep network is simple parts, stacked, that learn — with depth meaning a ladder of learned representations.Deep learning is not brain-like magic but a stack of simple mathematical steps that self-adjust. 'Deep' means many layers, each re-describing the input. Visualizations of trained image networks show a hierarchy from edges to objects that training carves out. The course's two questions: what is each rung, and how does the stack tune itself.
2. The Artificial NeuronDefine the artificial neuron (weighted sum + bias + activation) and why the nonlinearity is essential.A neuron multiplies inputs by weights, sums them with a bias, and applies a nonlinear activation (ReLU is the plain version that made deep nets trainable). All learned knowledge lives in the weights and biases. Without the nonlinearity, stacked linear layers collapse into one, so the activation is what makes depth add power.
3. Stacking Into a NetworkShow how neurons stack into layered networks and how learned features replaced hand-engineered ones.Neurons are wired into input, hidden, and output layers; data flows forward. Cybenko (1989) showed a wide enough hidden layer can approximate continuous functions on a bounded region, but depth is efficient. The key shift: raw data goes in and the layers discover useful features themselves.
4. Learning as Descending a HillExplain the loss function and (stochastic) gradient descent as descending a loss landscape.The loss is one number measuring how wrong the network is. Picture it as a landscape over all weight settings; gradient descent feels the local downhill direction and steps that way, sized by the learning rate. Stochastic gradient descent estimates the slope from small random batches for speed, and the sampling noise often helps escape shallow dips.
5. Backpropagation: The EngineExplain backpropagation as sending the output error backward via the chain rule to get every weight's gradient.The loss is measured only at the output, but blame is spread across all weights. Backpropagation, popularised by Rumelhart, Hinton, and Williams in 1986, pushes the error backward layer by layer using the chain rule. Forward pass, loss, backward pass, update — repeated — is training, and it lets any architecture be trained end to end.
6. Convolutional Networks: Built for ImagesExplain CNNs via locality and weight sharing, and AlexNet's 2012 ImageNet breakthrough.Dense networks need infeasibly many weights for images and relearn patterns per location. CNNs connect neurons to local patches and share a filter across the whole image, giving efficiency and a detector applied everywhere. The SuperVision/AlexNet 2012 win (15.3% vs 26.2% top-5 error) turned the field toward deep learning.
7. Sequences, Attention, and TransformersTrace sequence models from RNNs to self-attention and transformers, and why scale mattered.Sequences need long-range links, which RNNs handle poorly and slowly. Self-attention (2017, Attention Is All You Need) lets every position attend to every other directly and in parallel, enabling transformers. Scaling transformers works: GPT-3 (2020) had 175 billion parameters and was trained for 300 billion tokens — yet it is still neurons, nonlinearities, and backprop, arranged cleverly.
8. Why Now, and What Still BreaksExplain why deep learning succeeded when it did, its real limitations, and leave the reader with something to notice.Old ideas exploded in the 2010s once data, GPU compute, and unlocks like ReLU converged (AlexNet 2012). The same design brings real limits: data/compute/energy hunger, brittleness to crafted inputs, and opacity. The unifying idea: compose simple differentiable parts so error can flow backward through all of them.

Questions this course answers

Grounded in trusted sources

• LeCun, Bengio, Hinton — Deep learning, Nature 521 (2015): https://www.nature.com/articles/nature14539
• Rumelhart, Hinton, Williams — Learning representations by back-propagating errors, Nature 323 (1986): https://www.nature.com/articles/323533a0
• Krizhevsky, Sutskever, Hinton — ImageNet Classification with Deep Convolutional Neural Networks, NeurIPS 2012: https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks
• Vaswani et al. — Attention Is All You Need (2017): https://arxiv.org/abs/1706.03762
• Brown et al. — Language Models are Few-Shot Learners (GPT-3, 2020): https://arxiv.org/abs/2005.14165
• Cybenko — Approximation by superpositions of a sigmoidal function, Math. Control Signals Systems (1989): https://link.springer.com/article/10.1007/BF02551274
• Zeiler & Fergus — Visualizing and Understanding Convolutional Networks (2014): https://arxiv.org/abs/1311.2901
• Szegedy et al. — Intriguing properties of neural networks (2013): https://arxiv.org/abs/1312.6199

Every Wunder lesson is built from real, reputable sources — never invented.

Related Science courses