GPU × LLMAda Lovelace · 24GB VRAM

Best LLM Models for the NVIDIA RTX 4090

Best consumer GPU for local 7B–34B model inference

The RTX 4090's 24 GB GDDR6X and 1,008 GB/s bandwidth make it the fastest consumer GPU for local LLM inference. It runs Llama 3.1 8B at 100+ tokens/sec in BF16, Llama 3.3 70B in Q4 at 20–30 tokens/sec, and Mistral 7B at near-real-time speeds. For developers running local inference with llama.cpp, Ollama, or LM Studio, the 4090 is the default recommendation. Two 4090s (48 GB combined via llama.cpp split) can run 70B models in Q5 comfortably.

VRAM fit key: BF16 fit — runs in full precision INT4 fit — runs quantised API only — closed-weight, served via API

Recommended Models (24GB Ada Lovelace)

Common Use Cases

local inference7B–13B BF1670B Q4developer workstationsfine-tuning 7B

VRAM Sizing Guide for NVIDIA RTX 4090

BF16 (full precision): requires ~2 bytes per parameter. On 24GB VRAM, the maximum model size in BF16 is approximately 12288B parameters (leaving ~10% headroom for KV cache).

INT8 quantisation: halves memory to ~1 byte/param — fits up to ~24576B parameters. Minimal quality loss on most models.

INT4 / Q4 quantisation: ~0.5 bytes/param — fits up to ~49152B parameters. Recommended for 70B+ models on this GPU. Use GPTQ or AWQ for best quality.

KV cache: grows with context length and batch size. At 128K context with batch size 8, a 70B model in INT4 needs ~12GB additional VRAM for KV cache. Factor this into your serving configuration.