This repository contains a structured, educational sequence of Jupyter Notebooks detailing the mathematical, architectural, and training mechanics of modern Generative AI and Diffusion Models. The project walks through building foundational concepts from scratch using PyTorch, starting from basic convolutional neural networks (CNNs), to forward diffusion processes, to constructing a U-Net architecture with time embeddings, to training a Denoising Diffusion Probabilistic Model (DDPM), and finally implementing Latent Diffusion models, cross-attention mechanisms, and Parameter-Efficient Fine-Tuning (PEFT) using Low-Rank Adaptation (LoRA).
Understanding diffusion models (like Stable Diffusion or Imagen) is notoriously difficult due to the dense mathematical equations (Markov chains, variational lower bounds, and noise estimation) and complex architectures involved (residual UNet blocks, sinusoidal time embeddings, cross-attention matrices). Most online examples are either too simple (abstracting everything behind library pipelines) or too complex (spaghetti code inside large frameworks), making it hard to bridge the gap between theory and code.
This project solves this by:
- Writing Implementations From Scratch: Breaking down U-Net residual blocks, forward noise scheduling, and cross-attention components into clean, readable PyTorch modules.
- Step-by-Step Curriculum: Sequencing notebooks logically so that each topic builds directly on the previous one.
- Focusing on Core Mathematical Mechanics: Visualizing how the forward diffusion process degrades images and how the reverse process reconstructs them.
- Curriculum-Driven Learning: Serves as a self-paced textbook for deep learning practitioners wanting to master generative models.
- Architecture References: Provides clean, standalone implementations of custom U-Nets, time embeddings, Variational Autoencoders (VAEs), and Cross-Attention layers that can be copy-pasted and adapted for other tasks.
- LoRA Fine-tuning Template: Offers a fully functional notebook for fine-tuning text-to-image stable diffusion pipelines on a custom image dataset using PEFT.
- Goal: Refresher on standard Multi-Layer Perceptrons (MLPs) and Convolutional Neural Network (CNN) operations in PyTorch.
- Key Concepts: Feature maps, pooling layers, convolutional kernels, activation functions.
- Goal: Implementing the mathematical forward diffusion process (adding Gaussian noise under a specified scheduler).
-
Key Concepts: Beta schedules, alpha schedules, cumulative product (
alphas_bar), and the closed-form$q$ -sampling equation to sample noisy images at any arbitrary timestep$t$ without iterating.
- Goal: Constructing the U-Net architecture used to estimate the noise added to an image.
- Key Concepts: Residual Blocks, Time Embeddings (sinusoidal and linear projections), Group Normalization, and Skip Connections.
- Goal: Implementing the full DDPM training loop and loading saved model weights for image inference.
- Key Concepts: Loss calculation (MSE between target and predicted noise), model checkpoints, and sampling loops to denoise random Gaussian noise into generated outputs.
- Goal: Building and training a Variational Autoencoder (VAE) to encode inputs into a continuous latent space.
- Key Concepts: Latent space representation, reparameterization trick, Kullback-Leibler (KL) divergence loss, Convolutional Attention Blocks (CAB), and Channel Attention Layers (CALayer).
- Goal: Building the conditioning mechanisms that allow user prompts to guide the generation process.
- Key Concepts: Query, Key, and Value projections, Cross-Attention matrices, and text/class conditioning injection into U-Net intermediate layers.
- Goal: Implementing Low-Rank Adaptation (LoRA) to fine-tune a pre-trained Stable Diffusion pipeline on custom 3D icons.
- Key Concepts: Parameter-Efficient Fine-Tuning (PEFT), freezing original weights, rank decomposition matrices, Hugging Face Diffusers integration.
Install the required packages to run the notebooks:
pip install -r requirements.txt(Key dependencies include PyTorch, torchvision, Hugging Face diffusers & transformers, einops, scikit-learn, and accelerate)