You need to chunk your raw text (Project Gutenberg, FineWeb, or TinyStories) into fixed-context windows. If your context length is 256 tokens, you slide a window across your dataset. This prepares the input tensors (B, T) where B is batch size and T is sequence length. Pillar 3: The Architecture – Coding Attention (The "Self" Part) This is the heart of the PDF. You cannot copy-paste from PyTorch's nn.Transformer layer. You must build the Masked Multi-Head Attention from scratch using basic matrix multiplication ( torch.matmul ) and softmax.
A naive "character-level" tokenizer (treating each letter as a token) would require a context window of 10,000 steps for a short paragraph. A sub-word tokenizer reduces that to ~200 steps. build a large language model %28from scratch%29 pdf
import tiktoken enc = tiktoken.get_encoding("gpt2") text = "Hello, I am building an LLM." tokens = enc.encode(text) # Output: [15496, 11, 314, 716, 1049, 1040, 13] You need to chunk your raw text (Project