Why GPUs and CUDA? A beginner's starting point

Start with the problem

Imagine that you need to brighten one photograph.

The photograph contains millions of pixels, and the same small calculation must be applied to every pixel. A CPU can process those pixels, but it has only a small number of powerful cores. Most of the work must wait its turn.

A GPU takes a different approach. It has many smaller processing units that can work on many pixels at the same time. Instead of finishing one task as quickly as possible, it is designed to finish a large collection of similar tasks together.

flowchart LR
  subgraph CPU["CPU: a few powerful workers"]
      C1["Pixel 1"] --> C2["Pixel 2"] --> C3["Pixel 3"] --> C4["Pixel 4"]
  end

  subgraph GPU["GPU: many workers in parallel"]
      G["Start"] --> G1["Pixel 1"]
      G --> G2["Pixel 2"]
      G --> G3["Pixel 3"]
      G --> G4["Pixel 4"]
  end

  style CPU fill:#4a90d9,color:#fff
  style GPU fill:#e8744f,color:#fff

This difference is the reason GPU programming exists.

CPU and GPU have different strengths

A CPU is the general-purpose processor that runs your operating system and most application code. Its cores are good at complex decisions, changing instructions, and tasks where each step depends on the previous one.

A GPU is a helper processor built for throughput: completing a large amount of similar work. It works best when a problem can be divided into thousands of small pieces that can run mostly independently.

A GPU is not a faster CPU, and it does not replace one. Real programs use both: the CPU organizes the work, and the GPU handles the large parallel part.

What CUDA is

CUDA is NVIDIA’s platform for programming NVIDIA GPUs.

Without CUDA, developers would need to express general calculations through graphics APIs. CUDA lets you write a function called a kernel and run many copies of it on the GPU.

A typical CUDA program follows this simple pattern:

1. The CPU prepares some data.
2. The data is copied to GPU memory.
3. The CPU asks the GPU to run a kernel.
4. Many GPU threads process different pieces of the data.
5. The result is copied back to the CPU.

You do not need to understand kernels, threads, or GPU memory yet. The next articles introduce them one at a time.

When a GPU helps

A problem is a promising GPU candidate when:

• there is a lot of data;
• the same calculation is repeated many times;
• many pieces can be processed independently; and
• the calculation takes enough work to justify moving data to the GPU.

Image processing is a good example because each output pixel can often be calculated independently. Matrix operations are another: the same multiply-and-add pattern is repeated across many values.

When a GPU does not help

GPU acceleration has a cost. Data usually has to move between CPU memory and GPU memory, and launching GPU work has overhead.

A GPU may be the wrong choice when:

• the input is small;
• each step depends on the result of the previous step;
• the code makes many different decisions; or
• moving the data takes longer than processing it on the CPU.

The goal of CUDA programming is not to move everything to the GPU. It is to recognize the part of a program that has enough parallel work to benefit.

Why this series uses CUDA

CUDA is a practical place to learn GPU programming because it provides:

• a C/C++ programming model;
• mature debugging and profiling tools;
• optimized libraries for common calculations; and
• extensive use in scientific computing, graphics, and machine learning.

CUDA runs on NVIDIA GPUs, so it is not a vendor-neutral platform. However, the ideas you will learn—parallel threads, memory movement, synchronization, and performance trade-offs—also apply to other GPU programming systems.

What you will learn

This series starts with the story of how GPUs became programmable, then moves to your first CUDA kernel. From there, you will learn how threads are organized, where data lives, how threads coordinate, and how to measure and improve performance.

You only need basic C or C++ familiarity. No GPU programming experience is required.

What comes next

If you want a quick overview of other hardware such as TPUs, NPUs, FPGAs, and ASICs, read the optional article A simple guide to CPUs, GPUs, TPUs, NPUs, FPGAs, and ASICs.

Otherwise, continue directly to GPUs: from pixels to parallel supercomputers, which explains how graphics hardware evolved into the programmable processor used by CUDA.