GPU Performance Lab

GPUs are expensive, and AI runs on them. Teams that run their own models need developers who can make GPU code fast. That skill starts with CUDA. Take the first step on your phone: short CUDA puzzles, then labs on real cloud GPUs that work up to softmax, SwiGLU and attention, the kernels inside LLMs. LAB PATHS: FROM A CPU LOOP TO LLM KERNELS 20 labs in 4 paths, all open on the free plan: • Elementwise Kernels: your first CUDA kernel, grid-stride loops, launch shape and coalescing. Then "SwiGLU: Apply Fusion to a Transformer MLP": the activation stays in a register instead of a round trip through memory. • Reductions: from a CPU sum through atomics, a block tree and warp shuffles to "Stable Softmax, Two Row Reductions" and "Online Softmax, One Read Instead of Three", the trick modern attention kernels are built on. • Matrix Multiplication, the core math of Transformer models: from a CPU loop to a first CUDA kernel, then shared-memory tiling at N = 4096. • Decoder Block: a small CPU version of a modern decoder block (RMSNorm, QKV, RoPE, grouped-query attention, SwiGLU). Then a packed QKV projection with tiling, RoPE and grouped-query attention as layout instead of copies, and causal attention, correct first. The sizes are small, for teaching; there are no real model weights. HOW A LAB WORKS Put the kernel together from pieces: fill each hole with the right line, or pick the launch size. Check your answer, then run it on a real cloud GPU. The code compiles and runs there, not on your phone, so you don't need a GPU of your own. You get a report card: the GPU, the kernel time, the launch, and whether it met the lab's goals, such as a speedup or a limit on memory traffic. Then you answer a few questions about it. Labs need a free GPUFlight account: sign in with Apple, Google, GitHub or an email and password. The free plan includes 5 GPU runs a day and 500 credits a month for premium GPUs. THE NOTEBOOK: 14 SKILLS, 106 PUZZLES Each page asks one question: which element this thread works on, how many warps fit on the SM, which bank serves word 37. Pick an answer box or tap a line of code. Get it wrong and the page shows the right answer and why. • Threads & Blocks, Rows & Columns, The Warp: how threads find their work, in 1D and 2D, 32 at a time • Memory Lane: copies, 32-byte sectors and strided reads • First Kernel and Occupancy: vector add, its launch, and which resource runs out first • Branches: when the threads of one warp disagree • Memory Ladder and Shared Banks: registers, shared memory, caches, global memory and bank conflicts • Working Together: barriers, races, atomics and tree reductions • Copy & Overlap: streams, async copies and pinned memory • PTX 101, What PTX is, Optimizing with PTX: the same kernel closer to the hardware, fast math, and FP8 on tensor cores Every skill opens with a short moving lesson. The notebook works offline and needs no account. PRACTICE SUBSCRIPTION (OPTIONAL) Practice removes the daily run cap and raises your monthly credits for premium GPUs from 500 to 1,000. It is an auto-renewable monthly subscription; the price is shown in the app in your local currency. • Payment is charged to your Apple Account when you confirm the purchase. • The subscription renews automatically unless it is cancelled at least 24 hours before the end of the current period. Your account is charged for renewal within 24 hours before the end of the current period. • Manage or cancel it in your Apple Account settings (Settings > your name > Subscriptions), or from "Manage subscription" on the app's plan page. • Deleting your GPUFlight account does not cancel an App Store subscription. Terms of Use (EULA): https://www.apple.com/legal/internet-services/itunes/dev/stdeula/ Privacy Policy: https://app.gpuflight.com/privacy CUDA is a trademark of NVIDIA Corporation. GPU Performance Lab is not affiliated with or endorsed by NVIDIA.