GPU Benchmark Comparison — TFLOPS, Memory Bandwidth & Cloud Pricing Across 25+ GPUs
Compare TFLOPS, memory bandwidth, inference latency, and live cloud pricing side-by-side across 25+ GPU models. Select up to 6 GPUs.
How to read GPU benchmark numbers
FP16 TFLOPS (teraflops of 16-bit floating-point operations per second) is the headline compute metric for LLM training and inference. The H100 SXM delivers 989 TFLOPS FP16 vs the A100's 312 TFLOPS — a 3.2× raw advantage. However, real-world LLM throughput improvements are typically 2–2.5× due to memory bandwidth bottlenecks: transformer attention layers are memory-bound, not compute-bound, so raw FLOPS don't translate linearly. Memory bandwidth (GB/s) is often the more predictive metric for inference workloads. The H100's 3.35 TB/s HBM3 bandwidth vs the A100's 2 TB/s HBM2e is a 1.67× advantage that maps more directly to time-to-first-token improvements.
VRAM capacity and model fit
VRAM capacity determines which model sizes fit on a single GPU or require multi-GPU tensor parallelism. A Llama 3.1 7B model in FP16 requires approximately 14GB VRAM for weights alone — fitting on an RTX 4090 (24GB) with room for KV cache. Llama 3.1 70B requires 140GB in FP16 — requiring two A100 80GB GPUs or a single H100 NVL (94GB). Llama 3.1 405B requires 810GB — a minimum of 10× A100 80GB or 5× H100 SXM. Quantisation (INT8, INT4, GPTQ, AWQ) reduces VRAM requirements by 2–4× at a small quality cost, enabling larger models on smaller GPUs. The benchmark tool shows VRAM capacity for all 25 tracked GPU models.
TDP and data-centre power constraints
TDP (thermal design power) determines the power draw and cooling requirements for GPU deployments. The H100 SXM has a 700W TDP — a full 8-GPU DGX H100 system draws 10.2 kW. At $0.10/kWh, that's $730/month in electricity per node, adding 15–25% to the total cost of ownership versus the hourly cloud rate alone. For on-premises or colocation deployments, power density (W/rack unit) is often the binding constraint — modern GPU servers require 30–60 kW per rack, requiring specialised liquid cooling infrastructure. Cloud providers absorb these costs in their hourly rates, but understanding TDP helps explain why H100 on-demand rates are 3–4× higher than A100 rates despite a 3.2× compute advantage.
NVLink and multi-GPU scaling
NVLink is NVIDIA's high-bandwidth GPU interconnect that enables efficient tensor parallelism across multiple GPUs. NVLink 4.0 (H100) provides 900 GB/s bidirectional bandwidth per GPU — 1.5× faster than NVLink 3.0 on A100 (600 GB/s) and 9× faster than PCIe 4.0 (100 GB/s). For models that require tensor parallelism across GPUs (70B+ parameters), NVLink bandwidth directly limits training throughput: all-reduce communication during backpropagation consumes 10–30% of total training time on PCIe-connected GPUs, versus 3–8% on NVLink. Consumer GPUs (RTX 4090, RTX 3090) lack NVLink entirely — multi-GPU scaling relies on PCIe, making them impractical for large-scale training despite their competitive single-GPU performance.
MLPerf benchmarks vs synthetic TFLOPS
Synthetic TFLOPS figures (from NVIDIA product pages) measure peak theoretical throughput under ideal conditions. MLPerf Training benchmarks measure time-to-accuracy on standardised tasks (ResNet-50, BERT, GPT-3, Stable Diffusion) under realistic conditions. MLPerf v4.1 results show the H100 SXM training GPT-3 175B approximately 2.3× faster than A100 in equivalent cluster configurations — lower than the 3.2× raw TFLOPS ratio due to memory bandwidth bottlenecks and communication overhead. For inference, MLPerf Inference benchmarks measure queries-per-second and latency at 99th percentile — a more reliable predictor of production performance than TFLOPS alone. The benchmark data in this tool uses MLPerf v4.1 results where available, with synthetic TFLOPS as fallback.