GPU Benchmark Guide for Machine Learning: What the Numbers Actually Mean

GPU spec sheets are optimized for marketing, not for predicting real ML training or inference throughput. A GPU rated at 989 TFLOPS FP16 will rarely deliver anywhere near that figure on a real transformer training run — typical utilization is 30–60% of theoretical peak. Understanding why requires knowing which benchmarks matter, which don't, and how to translate spec-sheet numbers into actual tokens-per-second or samples-per-hour.

This guide breaks down the key GPU performance metrics for ML workloads — FLOPS, memory bandwidth, memory capacity, interconnect speed, and software stack maturity — and explains how they interact in practice. Use the GPU model comparison tool to compare specs side-by-side across 25+ GPUs.

FLOPS (floating-point operations per second) is the most cited GPU benchmark, but raw FLOPS is a poor predictor of ML performance on its own. The H100 SXM5 delivers 989 TFLOPS FP16 and 1,979 TFLOPS FP8. The A100 delivers 312 TFLOPS FP16. On paper, H100 is 3.2× faster. In practice, real LLM training throughput improvements are 2–2.5×.

The gap between theoretical and achieved FLOPS comes from three sources: (1) memory bandwidth bottlenecks — transformer attention layers are memory-bound, not compute-bound, so raw FLOPS don't help; (2) communication overhead — in multi-GPU training, all-reduce operations consume 10–30% of total time; (3) framework overhead — kernel launch latency, Python interpreter overhead, and data loading can consume 15–25% of GPU time. MLPerf Training benchmarks are the most reliable public source for real-world ML throughput — they measure time-to-accuracy on standardized tasks. See H100 vs A100 benchmark comparison for a full side-by-side.

For transformer models — the architecture underlying virtually all modern LLMs — memory bandwidth is the primary performance bottleneck, not compute. During inference, each forward pass reads the entire model weight matrix from HBM (high-bandwidth memory) into compute units. A 70B parameter model in FP16 requires reading ~140GB of weights per forward pass. At H100's 3,350 GB/s bandwidth, that's ~42ms of pure memory read time — before any computation.

Memory bandwidth benchmarks to know: H100 SXM5: 3,350 GB/s (HBM3). A100 SXM4: 2,039 GB/s (HBM2e). H100 PCIe: 2,000 GB/s. A100 PCIe: 1,935 GB/s. RTX 4090: 1,008 GB/s (GDDR6X). RTX 3090: 936 GB/s. The H100 SXM5's 1.6× bandwidth advantage over A100 SXM4 directly translates to ~1.6× faster inference for memory-bound workloads — a more reliable predictor than FLOPS for LLM serving. Check live H100 and A100 pricing to see whether the bandwidth premium justifies the cost for your workload.

GPU VRAM capacity determines which models you can run at all, and at what batch size. The rule of thumb for inference: a model requires approximately 2 bytes per parameter in FP16, or 1 byte per parameter in INT8/FP8. A 70B parameter model needs ~140GB in FP16 — requiring two H100 80GB GPUs (160GB total) or four A100 40GB GPUs (160GB total).

For training, memory requirements are 4–6× higher than inference due to optimizer states, gradients, and activations. Training a 7B model with AdamW in FP16 requires ~56GB minimum — fitting on a single H100 80GB but not on a single A100 40GB. Key VRAM capacities: H100 SXM5: 80GB HBM3. A100 SXM4: 80GB or 40GB HBM2e. H100 NVL: 94GB (dual-GPU NVLink module). RTX 4090: 24GB GDDR6X. RTX 3090: 24GB GDDR6X. A6000 Ada: 48GB GDDR6. Use the GPU spec comparison tool to filter GPUs by VRAM capacity for your target model size.

When a model doesn't fit on a single GPU, or when you want to increase batch size for training throughput, multi-GPU scaling becomes critical. The interconnect between GPUs determines how efficiently they can share data — and it's often the bottleneck in large-scale training.

NVLink 4.0 (H100): 900 GB/s bidirectional bandwidth per GPU pair. NVLink 3.0 (A100): 600 GB/s bidirectional. PCIe 4.0 (consumer GPUs): 64 GB/s bidirectional — 14× slower than NVLink 4.0. This matters enormously for tensor parallelism (splitting a single model layer across GPUs) and pipeline parallelism. A 4×H100 NVLink cluster can achieve near-linear scaling for tensor-parallel inference of 70B+ models. A 4×RTX 4090 PCIe cluster will see severe communication bottlenecks at the same model size — effective throughput may be only 2–2.5× a single GPU despite 4× the compute. For multi-GPU training clusters, compare CoreWeave NVLink cluster pricing vs Lambda Labs multi-GPU instances.

Modern ML training and inference use reduced-precision formats to increase throughput and reduce memory usage. Understanding which formats your GPU supports — and at what performance level — is essential for benchmarking.

FP32 (32-bit float): Used for optimizer states and loss scaling. All GPUs support it. H100: 67 TFLOPS, A100: 77.6 TFLOPS. FP16 (16-bit float): Standard training precision. H100: 989 TFLOPS, A100: 312 TFLOPS. BF16 (bfloat16): Preferred over FP16 for training stability — same range as FP32, lower precision. H100: 989 TFLOPS, A100: 312 TFLOPS. FP8 (8-bit float): Supported natively on H100 only. H100: 1,979 TFLOPS — doubles throughput vs FP16 with minimal accuracy loss for inference. INT8: Supported on H100, A100, and RTX 30/40 series via TensorRT. H100: 3,958 TOPS. INT4: Supported via GPTQ/AWQ quantization on most modern GPUs. Enables running 70B models on a single H100 80GB. For inference cost optimization, INT8 and INT4 quantization can reduce per-token costs by 2–4× vs FP16. See LLM inference pricing across providers to compare managed API costs against self-hosted quantized inference.

The most useful benchmarks for ML workloads are task-specific, not synthetic. For LLM training: tokens per second per GPU (measure with your actual model architecture and batch size). For LLM inference: tokens per second at your target batch size and latency percentile (p50, p95). For image model training: samples per second (images/sec) at your target resolution and batch size.

Reliable public benchmark sources: MLPerf Training (mlcommons.org) — time-to-accuracy on GPT-3, ResNet, BERT, and other standard models. MLPerf Inference — throughput and latency benchmarks for inference serving. Hugging Face LLM-Perf — open-weight LLM throughput across GPU types. Lambda Labs GPU Benchmark — practical training throughput for common model architectures. When evaluating a cloud provider, always run your own benchmark on a short trial instance before committing to a long reservation. Use the GPU price history charts to track whether rates have been stable before signing a long-term contract.

Here is a practical benchmark summary for the three most commonly deployed GPUs for ML workloads in 2026, based on real-world training and inference measurements:

H100 SXM5 80GB: FP16 TFLOPS: 989. Memory bandwidth: 3,350 GB/s. VRAM: 80GB HBM3. NVLink: 900 GB/s. Best for: Training >30B models, high-throughput inference serving, FP8 workloads. Typical LLM training throughput (Llama 3 70B, FP16): ~12,000 tokens/sec.

A100 SXM4 80GB: FP16 TFLOPS: 312. Memory bandwidth: 2,039 GB/s. VRAM: 80GB HBM2e. NVLink: 600 GB/s. Best for: Training 7B–30B models, cost-efficient inference, mature software ecosystem. Typical LLM training throughput (Llama 3 70B, FP16): ~5,200 tokens/sec.

RTX 4090 24GB: FP16 TFLOPS: 165. Memory bandwidth: 1,008 GB/s. VRAM: 24GB GDDR6X. PCIe only. Best for: Fine-tuning <13B models, inference of quantized models up to 70B (INT4), development and experimentation. Typical LLM training throughput (Llama 3 7B, BF16): ~3,800 tokens/sec.

Compare current cloud pricing for all three GPUs to calculate cost-per-token for your workload.