GPU Programming
Posts and notes about gpu programming.
Series & Posts
1
How a CPU actually works: cores, clocks, and execution
2 CPU caches and memory hierarchy: why memory access speed matters
3 CPU pipelines and instruction-level parallelism
4 Memory models and why concurrent CPU code is hard
5 SIMD and vectorization: parallelism on a single CPU core
6 Processes, threads, and context switching
1
Why GPUs and CUDA? A beginner's starting point
2 A simple guide to CPUs, GPUs, TPUs, NPUs, FPGAs, and ASICs
3 GPUs: from pixels to parallel supercomputers
4 Your first CUDA program: kernels, threads, and grids
5 Python GPU programming: CuPy vs Numba vs PyCUDA
6 Thread hierarchy in CUDA: threads, blocks, warps, and grids
7 Inside a modern NVIDIA GPU: SMs, schedulers, CUDA cores, and tensor cores
8 Warp divergence in CUDA: detection and optimization
9 Floating-point performance on GPUs: precision, FLOPs, and numerical error
10 CUDA memory hierarchy: where your data lives matters
11 Memory coalescing: the most important optimization you will learn
12 Shared memory and tiling: the key to fast matrix operations
13 Debugging and profiling CUDA programs
14 Device functions, host functions, and CUDA function qualifiers
15 Building reusable CUDA libraries with CMake and Python bindings
16 CUDA synchronization and atomics: __syncthreads, atomicAdd, and barriers
17 Parallel prefix sum and reduction: the core parallel primitives
18 Concurrent data structures on the GPU
19 CUDA streams and asynchronous execution
20 cudaEventSynchronize and CUDA events: accurate kernel timing
21 Dynamic parallelism: kernels launching kernels
22 Unified virtual memory: one pointer for CPU and GPU
23 Multi GPU CUDA: NCCL, NVLink, and peer access
24 Memory allocation patterns and multi-dimensional arrays in CUDA
25 Texture and constant memory: specialized caches
26 CUDA occupancy and register pressure: performance tuning guide
27 Case study: matrix multiplication from naive to cuBLAS speed
28 Case study: implementing a convolution layer in CUDA
29 Case study: reduction and histogram at scale
30 Heterogeneous computing: CPU and GPU working together
31 Advanced memory patterns: pinned memory, zero-copy, and more
32 Advanced stream patterns and concurrent kernel execution
33 Performance case studies and optimization patterns
34 CUDA Sobel edge detection: from naive kernel to profiled pipeline
35 Where to go from here: CUDA ecosystem and next steps