Best LLM Models for the NVIDIA L4
Low-power GPU for efficient 7B model serving at scale
The L4's 72W TDP makes it the most power-efficient data-centre GPU for LLM inference. At 24 GB VRAM it fits 7B–13B models in BF16 and runs Llama 3.1 8B at 40–60 tokens/sec. GCP L4 instances (g2-standard) are among the cheapest GPU instances available, making the L4 ideal for high-volume, latency-tolerant workloads like batch classification, embedding generation, and background summarisation. Its low power draw also makes it suitable for on-premise edge deployments.
Recommended Models (24GB Ada Lovelace)
Common Use Cases
VRAM Sizing Guide for NVIDIA L4
BF16 (full precision): requires ~2 bytes per parameter. On 24GB VRAM, the maximum model size in BF16 is approximately 12288B parameters (leaving ~10% headroom for KV cache).
INT8 quantisation: halves memory to ~1 byte/param — fits up to ~24576B parameters. Minimal quality loss on most models.
INT4 / Q4 quantisation: ~0.5 bytes/param — fits up to ~49152B parameters. Recommended for 70B+ models on this GPU. Use GPTQ or AWQ for best quality.
KV cache: grows with context length and batch size. At 128K context with batch size 8, a 70B model in INT4 needs ~12GB additional VRAM for KV cache. Factor this into your serving configuration.