📘 How do machines read language?

What you’ll learn

1. Text Preprocessing and TokenizationExplain how raw text is converted into model-ready tokens and integer IDs, including subword tokenization with BPE.This lesson covers the first stage of any deep-learning NLP pipeline: turning raw text into discrete tokens. It contrasts word-level tokenization and its out-of-vocabulary problem with subword methods like Byte-Pair Encoding, which merge frequent character pairs to keep vocabularies compact and coverage complete. It explains the trade-offs of vocabulary size on sequence length and model cost. Finally it shows how tokens map to integer IDs and special tokens that the network consumes.
2. Representing Words: From One-Hot to EmbeddingsDescribe the progression from sparse one-hot representations to dense word embeddings and what embeddings capture.This lesson motivates dense word embeddings by exposing the limits of one-hot vectors, which are sparse, high-dimensional, and treat all words as equally dissimilar. It introduces embeddings as low-dimensional vectors learned via the distributional hypothesis so that similar-context words have nearby vectors. It explains what embeddings capture, including similarity and analogical structure like king-man+woman approximately equals queen, while noting these are statistical tendencies that can encode bias. It distinguishes static from contextual embeddings and frames the embedding layer as a trainable lookup matrix.
3. Word Embeddings: Word2Vec and GloVeCompare Word2Vec and GloVe as methods for learning pretrained static word embeddings.This lesson examines the two classic embedding methods. Word2Vec is prediction-based, using skip-gram or CBOW over local context windows and negative sampling for efficient training. GloVe is count-based, factorizing a global word-word co-occurrence matrix so vector dot products approximate log co-occurrence probabilities. Both are unsupervised and yield dense vectors that capture similarity and analogy with broadly comparable performance. The lesson highlights the practical value of reusing pretrained vectors as an early form of transfer learning.
4. Sequence Models: RNNs and LSTMsExplain how RNNs and LSTMs process sequences and identify their structural limitations.This lesson introduces recurrent neural networks, which process tokens one at a time while carrying a hidden state, and explains the vanishing/exploding gradient problem that makes long-range learning hard. It presents LSTMs, whose cell state and forget/input/output gates stabilize information flow and capture longer dependencies, noting GRUs as a simpler variant. It identifies two enduring limits: sequential, non-parallel computation and remaining difficulty with very long-range dependencies. It also introduces the encoder-decoder architecture and its fixed-vector bottleneck.
5. Attention and the Rise of TransformersExplain the attention mechanism and how Transformers use self-attention to supersede recurrent models.This lesson introduces attention as a way to let models focus on relevant input parts, removing the fixed-vector bottleneck of encoder-decoder RNNs. It frames attention with queries, keys, and values, and introduces self-attention, where each token attends to every other token in the same sequence. It explains that the Transformer removes recurrence entirely, processes positions in parallel, and adds positional encodings for order. It concludes by explaining why parallelism plus long-range modeling drove the shift to Transformers, leaving detailed mechanics to a separate course.
6. Core NLP Tasks and EvaluationIdentify core NLP tasks and select the correct evaluation metric for each.This lesson surveys core NLP tasks: classification including sentiment, token-level NER, machine translation, question answering, and generation. It pairs tasks with correct metrics: accuracy and F1 (precision and recall via their harmonic mean) for classification, with F1 preferred under class imbalance. It defines BLEU as an n-gram-overlap metric with a brevity penalty for machine translation, and perplexity as the exponentiated average negative log-likelihood for language models, where lower is better. It stresses matching the metric to the task to avoid misleading evaluation.
7. Mastery Quiz: NLP with Deep LearningIntegrate the course concepts and correctly apply evaluation metrics and architectural reasoning across NLP scenarios.This final lesson consolidates the full course arc: from tokenization and embeddings through RNNs/LSTMs to attention and Transformers, plus core tasks and evaluation. It emphasizes two mastery themes: matching the right metric (F1 for imbalanced classification, BLEU for translation, perplexity for language models) and understanding why attention-based models replaced recurrence. The mastery quiz applies these ideas to scenario questions. It serves as a synthesis and self-check before completing the module.

Questions this course answers

Grounded in trusted sources

• Dan Jurafsky and James H. Martin, Speech and Language Processing — tokenization, embeddings, sequence models
• Tomas Mikolov et al., “Efficient Estimation of Word Representations in Vector Space” (Word2Vec)
• Ashish Vaswani et al., “Attention Is All You Need” — Transformer architecture
• Hugging Face NLP Course — practical transformers overview, https://huggingface.co/learn/nlp-course
• Stanford CS224N course notes — embeddings, RNNs, attention

Every Wunder lesson is built from real, reputable sources — never invented.

Related courses