Best LLM Models for the AMD Instinct MI300X
Highest memory capacity GPU for MoE and 671B parameter models
The MI300X's 192 GB HBM3 is the largest VRAM pool of any single GPU in production, enabling DeepSeek R1 671B in BF16 on a single card — something no NVIDIA GPU can match without NVLink. Its 5,300 GB/s bandwidth also makes it the fastest for memory-bound inference workloads. ROCm support has matured significantly with vLLM, llama.cpp, and SGLang all supporting MI300X natively. Cloud availability through Azure ND MI300X instances and Vultr makes it increasingly accessible.
Recommended Models (192GB CDNA 3)
Common Use Cases
VRAM Sizing Guide for AMD Instinct MI300X
BF16 (full precision): requires ~2 bytes per parameter. On 192GB VRAM, the maximum model size in BF16 is approximately 98304B parameters (leaving ~10% headroom for KV cache).
INT8 quantisation: halves memory to ~1 byte/param — fits up to ~196608B parameters. Minimal quality loss on most models.
INT4 / Q4 quantisation: ~0.5 bytes/param — fits up to ~393216B parameters. Recommended for 70B+ models on this GPU. Use GPTQ or AWQ for best quality.
KV cache: grows with context length and batch size. At 128K context with batch size 8, a 70B model in INT4 needs ~12GB additional VRAM for KV cache. Factor this into your serving configuration.