GPU vs TPU vs NPU vs FPGA vs ASIC: choosing an AI accelerator

Before you write a single kernel

Every article after this one in the series assumes you already chose a GPU. That choice is not automatic. “Accelerator” today spans general-purpose CPUs, massively parallel GPUs, matrix-multiply ASICs sold as TPUs, on-device NPUs baked into phone and laptop chips, reconfigurable FPGAs, and fully custom silicon. Each one trades generality for efficiency in a different place, and the wrong choice costs you months of engineering time or a very large cloud bill.

This article is a map, not a tutorial. It has no CUDA code, because the point is to understand why you would reach for CUDA (and a GPU) instead of one of the alternatives. Read it once before you start the series, and come back to it whenever you are deciding what hardware a new project should target.

Six kinds of silicon

At the highest level, every accelerator sits somewhere on a line from “runs any program” to “runs exactly one computation, extremely fast.”

graph LR
  CPU["CPU
fully general
low throughput/watt"] --> GPU["GPU
data-parallel
general numeric kernels"]
  GPU --> TPU["TPU-class ASIC
fixed matmul pipeline
vendor toolchain"]
  GPU --> NPU["NPU
fixed inference pipeline
edge power budget"]
  GPU --> FPGA["FPGA
reconfigurable logic
custom fixed-function"]
  TPU --> ASIC["Custom ASIC
one computation
highest efficiency"]
  NPU --> ASIC
  FPGA --> ASIC
  style CPU fill:#4a90d9,color:#fff
  style GPU fill:#e8744f,color:#fff
  style ASIC fill:#9b59b6,color:#fff

• CPU. A handful of powerful cores, deep out-of-order pipelines, large caches, branch prediction. Optimized for latency on a single thread and for arbitrary, branchy, sequential logic. Terrible at throughput on data-parallel numeric work because most of the transistor budget goes to control logic, not arithmetic units.
• GPU. Thousands of simple arithmetic lanes, a SIMT execution model, and a memory system built for bandwidth rather than latency. Still fully programmable: any numeric kernel you can express as independent (or mostly independent) per-element work can run on it. This generality is exactly what the rest of this series is about.
• TPU (Tensor Processing Unit). A product category, not a technical term with one fixed definition; in this series it refers to purpose-built accelerators (most visibly Google’s) organized around a systolic array that streams matrix multiplies through a fixed hardware pipeline. Excellent at the dense matmuls that dominate deep learning, tied to a specific vendor toolchain and cloud, and less flexible for irregular or non-matmul-shaped compute.
• NPU (Neural Processing Unit). Also a vendor-specific product class, not a single spec: the name shows up on mobile SoCs, laptop chips, and camera/IoT silicon (Apple’s Neural Engine, Qualcomm’s Hexagon NPU, various Arm Ethos designs, and others). Built for low-power, low-latency inference of a constrained set of neural network operators, usually at INT8 or lower precision, with a fixed power and thermal envelope.
• FPGA (Field-Programmable Gate Array). An array of configurable logic blocks and routing that you program at the hardware level using an HDL (Verilog/VHDL) or high-level synthesis (HLS) tools. You are not writing a program that runs on fixed logic; you are describing new logic. That buys you bit-level customization and reconfigurability after deployment, at the cost of a much longer and more specialized development cycle than software.
• Custom ASIC (Application-Specific Integrated Circuit). Silicon designed and fabricated for exactly one computation, with every transistor devoted to that job. Highest possible efficiency per operation, but only if the workload is stable enough to justify a design and fabrication cycle that runs into the tens of millions of dollars and a year or more of lead time before the first chip exists.

The axes that actually matter

Comparing accelerators on a single “speed” number is close to meaningless. The choice is a multi-dimensional trade-off. These are the axes worth reasoning about explicitly.

Generality vs. efficiency

The more general a device is, the more of its silicon area and power budget goes toward flexibility (instruction decode, branch handling, reconfigurable routing) instead of raw arithmetic. A CPU can run anything but wastes most of its die on control logic for data-parallel numeric work. An ASIC devotes essentially all of its die to the one computation it was built for. GPUs, TPUs, NPUs, and FPGAs sit at different points between these extremes, and that position is a design decision, not an accident.

Latency vs. throughput

A single CPU thread finishes one unit of work as fast as physically possible; that is a latency-optimized design. A GPU finishes millions of units of work per second by running many of them simultaneously, but any individual unit of work may take longer to complete than it would on a fast CPU core, because it is waiting in a queue behind thousands of siblings. TPUs push this further: they are throughput machines almost exclusively, tuned for large batched matmuls, not for a single request that needs an answer in microseconds. NPUs invert the priority again, because on-device inference (a camera frame, a voice command) usually cares more about single-request latency and power than about aggregate throughput.

Training vs. inference

Training a large model requires forward and backward passes, gradient accumulation, optimizer state, and numerically stable higher-precision accumulation; it also benefits enormously from a mature, general-purpose programming ecosystem because model architectures change constantly during research. Inference, especially at the edge, is a fixed, already-validated computation graph that you want to run as cheaply and quickly as possible, often after quantizing to INT8 or lower. This is why GPUs and TPU-class accelerators dominate training, while NPUs and inference-specialized ASICs dominate high-volume deployed inference.

Architecture and memory system

The compute units are only half the story. A CPU has deep cache hierarchies tuned for locality and pointer-chasing. A GPU has wide, high-bandwidth memory (GDDR or HBM) and a memory hierarchy built to hide latency across many threads, not to minimize it for one. A TPU’s systolic array is built so that data flows through a grid of multiply-accumulate cells with minimal memory traffic between them, which is what lets it hit very high utilization on large matmuls specifically. An FPGA’s memory system is whatever you route it to be, which is powerful but entirely your responsibility to design.

Numeric formats

Training generally wants FP32 or mixed FP16/BF16 with FP32 accumulation for numerical stability. TPUs are historically strongly associated with BF16 (and its own reduced-precision matmul formats) because Google’s own hardware and software stack co-designed around it. NPUs overwhelmingly run INT8 or INT4 quantized inference to fit power and memory budgets. FPGAs and ASICs can use whatever custom format the designer chooses, including formats no general-purpose processor supports at all. The floating-point performance article later in this series goes deep on what these formats cost you numerically on a GPU.

Toolchain and ecosystem

A GPU running CUDA gets you the largest, most mature accelerated-computing ecosystem in existence: compilers, profilers, math libraries, and virtually every deep learning framework as a first-class target. A TPU ties you to a narrower, vendor-controlled toolchain (historically XLA-based compilation through frameworks like JAX and TensorFlow) that is excellent within its lane and inflexible outside it. NPUs usually ship with a vendor SDK that converts a trained model into the NPU’s fixed operator set; anything the SDK cannot map to a supported operator silently falls back to the CPU or GPU, often with a large performance cliff. FPGA development means HDL or HLS toolchains with compile (“synthesis and place-and-route”) times measured in hours, not seconds. ASIC development means full custom or semi-custom chip design flows that most software teams will never touch directly.

Power, cost, and portability

Power envelopes range from single-digit watts (embedded NPUs) to hundreds of watts per GPU in a data center rack. Non-recurring engineering cost is near zero for GPU software (you pay for compute time), moderate for FPGA designs (engineering time plus the board), and enormous for a custom ASIC (mask sets and fabrication runs). Portability follows the same pattern in reverse: CUDA code is portable across GPU generations and, with effort, across vendors through translation layers; an ASIC is portable to exactly the chip it was designed for and nothing else.

Comparison matrix

A decision flow

flowchart TD
  A["What are you building?"] --> B{"Is the workload
stable and extremely
high volume?"}
  B -- "No, still evolving / research" --> C{"Need general
programmability and
a mature ecosystem?"}
  C -- Yes --> GPU["GPU + CUDA/ROCm"]
  C -- No, tied to one cloud stack --> TPU["TPU-class ASIC"]
  B -- "Yes, and it will not change for years" --> D{"Volume high enough to
justify NRE and fab cost?"}
  D -- Yes --> ASIC["Custom ASIC"]
  D -- "No, but need custom logic now" --> FPGA["FPGA"]
  A --> E{"Running on a battery/
thermally constrained device?"}
  E -- Yes --> NPU["NPU (or CPU fallback)"]
  style GPU fill:#4a90d9,color:#fff
  style TPU fill:#e8744f,color:#fff
  style ASIC fill:#9b59b6,color:#fff
  style FPGA fill:#2ecc71,color:#fff
  style NPU fill:#f1c40f,color:#000

This flow is deliberately simplified. In practice, real systems mix accelerators: a phone app trains nothing on-device but runs inference on its NPU with a CPU fallback; a research lab trains on GPUs and may later move a stable, high-volume inference workload to a TPU-class accelerator or a custom ASIC once the model architecture stops changing.

Deployment context changes the answer

The same workload gets a different recommendation depending on where it runs.

• Data center. GPUs and TPU-class accelerators both live here. GPUs win when you need flexibility across many model architectures, custom kernels, or a broad software ecosystem. TPU-class accelerators win when you are fully committed to one vendor’s stack and running large, stable, batched matmul-heavy training or serving jobs at massive scale.
• Workstation. A single GPU with a full CUDA (or ROCm) toolchain is the default for local development, small-scale training, and interactive experimentation. This is the environment the rest of this series assumes.
• Mobile. NPUs dominate for on-device inference because of the power budget; the GPU on the same SoC is a fallback for operators the NPU cannot execute, and the CPU is the fallback of last resort.
• Embedded. Power, cost, and certification constraints often push toward NPUs for inference or FPGAs when the logic needs to be deterministic, low-latency, or field-updatable without replacing hardware.
• Cloud. All of the above are available as rented capacity. The economic calculation shifts from “what can I afford to own” to “what is the cheapest way to rent enough throughput,” which changes the answer even for an identical workload.

Matching workload shapes to hardware

In practice

Most teams do not choose one accelerator forever. A realistic project lifecycle looks like: prototype and train on GPUs because the ecosystem and flexibility de-risk the research; once the model architecture is stable, evaluate whether a TPU-class accelerator or a custom inference ASIC is cheaper at your serving volume; and for anything shipping on a device, quantize and target the device’s NPU with a CPU/GPU fallback path for unsupported operators.

The GPU’s role in this lifecycle is disproportionate: it is almost always the first accelerator you touch, because it is the only one on this list that is simultaneously fast, general-purpose, and backed by a toolchain mature enough to build and debug a new idea in hours rather than weeks. That is why this series is about CUDA specifically, and why the next article starts from the GPU’s own history rather than jumping straight into TPUs, NPUs, or FPGAs.

What comes next

You now have a map of the accelerator landscape and a vocabulary (generality vs. efficiency, latency vs. throughput, training vs. inference) that will recur throughout this series whenever a design choice is really a hardware trade-off in disguise.

The next article, GPUs: from pixels to parallel supercomputers, narrows the focus to the GPU specifically: how a fixed-function graphics chip became a general-purpose parallel computer, why CUDA became the dominant programming model, and what a modern GPU looks like from 10,000 feet before you write your first kernel.