A10G

The NVIDIA A10G is an Ampere-generation data center GPU designed for inference serving, built on the GA102 die (Samsung 8nm). At just 150W TDP, it is one of the most power-efficient data center GPUs available, enabling dense rack deployments without specialized cooling. It delivers 125 TFLOPS of FP16 throughput and 250 TOPS of INT8, making it well-suited for serving quantized models at scale. It is the primary GPU in AWS g5 instances.

The 24GB GDDR6 configuration provides 600 GB/s of memory bandwidth — lower than HBM-based alternatives but higher than the T4's 320 GB/s. This fits models up to approximately 12B parameters at FP16, or 24B at INT8. The A10G uses PCIe 4.0 with no NVLink, so VRAM cannot be pooled across cards. Its 31.2 TFLOPS FP32 is notably high for a data center GPU, making it well-suited for mixed AI inference and graphics rendering workloads.

The A10G is the right choice for inference serving of models up to 13B parameters, particularly on AWS where it is the primary GPU in g5 instances. It is well-suited for Stable Diffusion and image generation workloads, mixed compute and graphics environments, and high-density inference deployments where power efficiency matters. It is not suitable for training large models (limited VRAM, no NVLink) or for memory-bound inference of 70B+ models where HBM bandwidth matters.