Unit 11 of 15

Text, Sequences, and Modern Generative Models

  • 6 lessons
  • Videos 72 to 77

Text and other sequences need representations of words and their order. This unit goes from sparse word counts to recurrent networks, attention and transformers, and ends with generative models.

Tokenization and TF-IDF weights turn documents into sparse vectors for naive Bayes and logistic regression, with splits that keep duplicates and sources apart. Recurrent networks carry a hidden state along a sequence and learn by backpropagation through time, and LSTM cells add gates. Scaled dot-product attention mixes value vectors with weights from queries and keys, and masks hide unavailable positions. Transformer blocks add positional information, residual connections and normalization. The last lesson compares autoregressive, variational, adversarial and diffusion models.

Every lesson will be one video. The videos are coming soon, and each lesson page already has the definitions and an example for every key term.

Sparse Text Features and Classification

  1. Text Features and TF-IDF

    How tokenization and a vocabulary fitted on training documents turn text into bag-of-words counts, how document frequency gives TF-IDF weights, and why the document-term matrix is stored as a sparse matrix.

    Video 72

  2. Text Classification Pipelines

    How a text classification pipeline fits multinomial naive Bayes or regularized logistic regression on sparse counts, and how text leakage and collection artifacts can make a text model look better than it is.

    Video 73

Recurrence, Attention and Transformers

  1. Sequence Models and Recurrent Neural Networks

    How a recurrent neural network carries a hidden state along a sequence, how backpropagation through time trains its shared weights, what LSTM and GRU gates do, and how padding, masks and teacher forcing are handled.

    Video 74

  2. Attention, Queries, Keys, and Values

    How scaled dot-product attention compares query and key vectors, turns the scores into attention weights by a row-wise softmax and averages value vectors, and what attention masks, causal masks and multi-head attention add.

    Video 75

  3. Transformers and Self-Supervised Language Models

    How token embeddings, positional encoding, layer normalization and residual connections form a transformer block, and how next-token prediction in a decoder-only model differs from masked language modeling.

    Video 76

Generative Models

  1. Generative Models and Their Evaluation

    What a generative model is, how autoregressive factorization gives an exact likelihood, and what latent-variable models, variational autoencoders and their evidence lower bound, adversarial networks and diffusion models optimize.

    Video 77