Large Language Model from Scratch

Go from tensors and gradients all the way to a working LLaMA-style decoder. This teaching lab covers PyTorch fundamentals, every transformer component (tokenisation, attention, normalisation, FFN), efficiency techniques (FlashAttention, MoE, LoRA), training pipelines, and inference optimisation — all designed to run on AMD GPUs.

Goals

• Master PyTorch fundamentals: forward pass, backpropagation, and training loops
• Understand every component inside a Transformer block (RoPE, RMSNorm, MHA, SwiGLU)
• Apply efficiency techniques: FlashAttention, MoE, LoRA, and quantization
• Build complete training and inference pipelines with KV-Cache optimisation
• Assemble a Tiny LLaMA from scratch as a capstone project

Foundations (LLM01 to LLM03)

• LLM01 — LLM Introduction and Inference

  Load pre-trained models, tokenize text, and run autoregressive text generation.

• LLM02 — Forward Propagation

  Tensor operations, linear layers, activations, and the building blocks of the forward pass.

• LLM03 — Backpropagation and Autograd

  Computation graphs, gradient verification, and complete training loops.

Transformer Components (LLM04 to LLM06)

• LLM04 — Tokenization and Positional Encoding

  Tokenizers, embeddings, sinusoidal encoding, and Rotary Position Embedding (RoPE).

• LLM05 — Normalization and FFN

  Layer norm, RMS norm, SwiGLU FFN, and Pre-LN vs. Post-LN comparisons.

• LLM06 — Attention Mechanisms

  Scaled dot-product attention, causal masking, MHA, and Grouped Query Attention (GQA).

Efficiency and Adaptation (LLM07 to LLM09)

• LLM07 — FlashAttention

  GPU memory hierarchy, tiling, and IO-aware attention optimisation.

• LLM08 — MoE and Numerical Precision

  Mixture of Experts architecture and floating-point / quantization formats.

• LLM09 — LoRA Fine-Tuning

  Parameter-efficient adaptation with Low-Rank Adaptation layers.

Training and Data (LLM10 to LLM11)

• LLM10 — Data Processing and Model Packaging

  Dataset preprocessing, state_dict, safetensors, and HuggingFace model wrapping.

• LLM11 — Model Training Pipeline

  Collate functions, loss functions, AMP, gradient accumulation, and HuggingFace Trainer.

Inference and Serving (LLM12 to LLM13)

• LLM12 — Inference and KV-Cache

  Prefill vs. decode, decoding strategies, and KV-Cache implementation.

• LLM13 — Sparse Attention and PagedAttention

  Sliding window attention, attention sinks, and block-based KV memory management.

Capstone (LLM14)

• LLM14 — Build a Tiny LLaMA from Scratch

  Full model implementation with RoPE, RMSNorm, MHA, SwiGLU, training, and text generation.

Explore the other teaching labs: Computer Vision, Deep Learning, Physics Simulation.