GPU × LLMAmpere · 24GB VRAM

Best LLM Models for the NVIDIA A10G

Cost-effective cloud GPU for 7B–13B model serving

The A10G is AWS's primary LLM inference GPU (g5 instances), available at $1–2/hr on-demand. Its 24 GB VRAM fits Llama 3.1 8B in BF16, Mistral 7B, and Llama 3.2 11B Vision comfortably. For teams on AWS deploying budget-tier models via SageMaker or EC2, the A10G is the most accessible path to GPU inference. Throughput is lower than the L40S or 4090, but the managed AWS ecosystem and spot pricing make it attractive for intermittent workloads.

VRAM fit key: BF16 fit — runs in full precision INT4 fit — runs quantised API only — closed-weight, served via API

Recommended Models (24GB Ampere)

Common Use Cases

AWS g5 instances7B–13B servingvision modelsbudget inference

VRAM Sizing Guide for NVIDIA A10G

BF16 (full precision): requires ~2 bytes per parameter. On 24GB VRAM, the maximum model size in BF16 is approximately 12288B parameters (leaving ~10% headroom for KV cache).

INT8 quantisation: halves memory to ~1 byte/param — fits up to ~24576B parameters. Minimal quality loss on most models.

INT4 / Q4 quantisation: ~0.5 bytes/param — fits up to ~49152B parameters. Recommended for 70B+ models on this GPU. Use GPTQ or AWQ for best quality.

KV cache: grows with context length and batch size. At 128K context with batch size 8, a 70B model in INT4 needs ~12GB additional VRAM for KV cache. Factor this into your serving configuration.