5 ms·
Hi internet friends, I recorded a workshop about building your own LLM without any math / ML prerequisites. It covers everything from machine learning fundament
by JustinAngel 4mo ago
Hi internet friends, I recorded a workshop about building your own LLM without any math / ML prerequisites. It covers everything from machine learning fundamentals, deep neural networks, transformer architecture, and pre/post-training.
The only prerequisite is being comfortable with learning through code & excel examples.
1. Sampling Large Language Models
https://go.justinangel.ai/video-1 https://go.justinangel.ai/video-1
2. Reverse Engineering Large Language Model
https://go.justinangel.ai/video-2 https://go.justinangel.ai/video-2
3. Perceptrons: wx+b
https://go.justinangel.ai/video-3 https://go.justinangel.ai/video-3
4. Activation Functions: ReLU, GELU, SwiGLU
https://go.justinangel.ai/video-4 https://go.justinangel.ai/video-4
5. GPU Coding: PyTorch, torch.compile(), fused kernels, CUDA, Triton
https://go.justinangel.ai/video-5 https://go.justinangel.ai/video-5
6. MLPs/FFNs: Multi-input, Multi-Layer Perceptrons, Feed-Forward Networks
https://go.justinangel.ai/video-6 https://go.justinangel.ai/video-6
7. Loss Functions: Residual errors, RMSE, Cross Entropy, Loss Landscapes
https://go.justinangel.ai/video-7 https://go.justinangel.ai/video-7
8. Backpropagation: Training loops, Optimizers, Learning Rate, Batch Size
https://go.justinangel.ai/video-8 https://go.justinangel.ai/video-8
9. Saving & Loading Models
https://go.justinangel.ai/video-9 https://go.justinangel.ai/video-9
10. Initialization: Kaiming, Glorot
https://go.justinangel.ai/video-10 https://go.justinangel.ai/video-10
11. Residuals: Addition, Scaling, Gated, Concatenation
https://go.justinangel.ai/video-11 https://go.justinangel.ai/video-11
12. Normalization: Pre-norm vs. Post-norm, RMSNorm, BatchNorm, LayerNorm
https://go.justinangel.ai/video-12 https://go.justinangel.ai/video-12
13. Regularization: Dropout, Gradient Clipping, Weight Decay
https://go.justinangel.ai/video-13 https://go.justinangel.ai/video-13
14. SoftMax
https://go.justinangel.ai/video-14 https://go.justinangel.ai/video-14
15. Tokenizers: By Character, By Word, BPE, SentencePiece
https://go.justinangel.ai/video-15 https://go.justinangel.ai/video-15
16. Embeddings: Absolute vs. Learned, Sinusoidal vs. RoPE
https://go.justinangel.ai/video-16 https://go.justinangel.ai/video-16
17. Attention: MHA, GQA, MQA, MLA
https://go.justinangel.ai/video-17 https://go.justinangel.ai/video-17
18. Transformers
https://go.justinangel.ai/video-18 https://go.justinangel.ai/video-18
19. Pre-training: Data Sources, Datasets, HTML Cleaning, Quality Filtering, Sharding
https://go.justinangel.ai/video-19 https://go.justinangel.ai/video-19
20. Evaluation: Leaderboards, Benchmarks, Verifiers vs LLM-as-Judge
https://go.justinangel.ai/video-20 https://go.justinangel.ai/video-20
21. Instruction Tuning: Alpaca & Other Formats, Self Instruct, Capabilities
https://go.justinangel.ai/video-21 https://go.justinangel.ai/video-21
22. Reinforcement Learning: Policy Optimization, SimPO
https://go.justinangel.ai/video-22 https://go.justinangel.ai/video-22
23. What We Didn't Cover: Scaling
https://go.justinangel.ai/video-23 https://go.justinangel.ai/video-23
Each section has slides teaching the concepts, followed by excel-by-hand developing intuition for the math, and then coding examples. The goal is able to grok all parts of modern LLM development.
We did this workshop in-person in San Francisco last month and hopefully the spaciousness of watching online works for everyone.
https://emilyhk.com/llm-workshop/ https://emilyhk.com/llm-workshop/
If don't like watching videos, you can get the slides and exercises and work self-paced.
https://go.justinangel.ai/deck https://go.justinangel.ai/deck