H100 80GB

The NVIDIA H100 80GB is the most capable widely-available GPU for large language model training and high-throughput inference as of 2025. Built on the Hopper architecture (GH100 die, TSMC 4N process), it delivers 1,979 TFLOPS of FP16 and BF16 throughput — a 6× improvement over the A100 in raw transformer compute. The headline feature is native FP8 support via the Transformer Engine: at 3,958 TFLOPS FP8, the H100 can nearly double throughput for inference workloads that tolerate reduced precision, with automatic scaling to maintain accuracy.

Memory configuration is 80GB of HBM3 with 3,350 GB/s bandwidth — 65% higher than the A100 80GB's 2,039 GB/s. This bandwidth advantage is the primary reason H100 outperforms A100 by more than the raw TFLOPS ratio suggests for memory-bound inference: each autoregressive token generation reads the full model weight matrix, so bandwidth directly determines tokens-per-second. The SXM5 form factor connects via NVLink 4.0 at 900 GB/s bidirectional, enabling near-linear tensor-parallel scaling across 8-GPU DGX H100 nodes for 70B+ model inference.

The H100 80GB is the right choice when training models above 30B parameters, running 70B+ inference at production throughput, or working with FP8 quantization pipelines. It is overkill for inference-only workloads on models below 30B parameters — the A100 80GB or L40S typically delivers better cost-per-token at those scales. At $2.50–$5.00/hr on-demand across major providers, the H100 commands a significant premium; validate that your workload is actually compute-bound (not memory-bandwidth-bound) before choosing it over cheaper alternatives.