Notes from the ML systems I build

A fused RMSNorm kernel in Triton, and what its bandwidth number actually means

A from-scratch kernel for the normalization in Llama, Qwen, Mistral and Gemma: forward and backward each fused into a single launch and wired into PyTorch autograd. ~6× over eager, bandwidth bound at 45% of an H100’s HBM3 peak.