Where the course pivots fully into modern NLP. The progression mirrors the historical arc that led to Transformers: character-level text generation → sentiment classification → sequence-to-sequence translation → attention. Each step reveals a limitation of the previous approach, motivating the next.

Concepts You’ll Learn About

  • Recurrent neural networks — hidden state as memory; unrolling through time; the vanishing gradient problem
  • LSTMs — gating mechanisms that let the network learn what to remember and what to forget
  • Sequence-to-sequence — encoder compresses the input into a context vector; decoder generates the output token by token
  • Beam search — keeping the top-k candidates at each decoding step instead of greedily picking one
  • Bahdanau attention — letting the decoder look back at all encoder states, not just the final one; the idea that becomes Transformers
  • Embeddings — learned dense representations of tokens; the input layer of every modern NLP model

Topics

  • RNNs and character-level text generation — Karpathy’s “Unreasonable Effectiveness of RNNs” as context, then generating Shakespeare one character at a time.

  • Reading day — working through assigned background reading on sequence models.

  • Sentiment analysis — IMDB movie review classification with an LSTM; how a recurrent network reads a sentence and produces a label.

  • Neural machine translation — starter — build an LSTM seq2seq translator from scratch; train on an English/Spanish sentence pair dataset.

  • Improve + round-trip translation — extend the starter with round-trip evaluation (English → Spanish → English); add attention and beam search from the reference notebook. Translation test cases Assignment: implement attention, run round-trip tests, submit with analysis of where the model succeeds and fails.

Notes

What’s next

Unit 13 replaces the recurrence entirely with self-attention — the Transformer — then uses it for translation, text generation, image captioning, and generative modeling.