T4

The NVIDIA T4 is a Turing-generation data center GPU designed for high-density inference serving. At just 70W TDP — the lowest of any data center GPU in widespread use — it enables extremely dense rack deployments: a single 1U server can host multiple T4 cards. Released in 2018, it delivers 65 TFLOPS of FP16 throughput and 130 TOPS of INT8, making it well-suited for serving quantized models at scale. It is the most widely available low-cost GPU on Google Cloud Platform and AWS.

The 16GB GDDR6 configuration provides 320 GB/s of memory bandwidth — the lowest of any current data center GPU. This limits the T4 to models up to approximately 7B parameters at FP16, or 16B at INT8. The low bandwidth makes it unsuitable for memory-bound autoregressive inference of larger models, but for batch inference of 7B models and smaller, the low cost per hour often compensates. The T4 uses PCIe 3.0 with no NVLink and lacks BF16 hardware acceleration.

The T4 is the right choice for high-volume, cost-sensitive inference of models up to 7B parameters — particularly when deployed in large quantities. It is the most cost-effective GPU for serving quantized BERT, DistilBERT, and similar encoder models at scale. It is not suitable for training, for models above 7B at FP16, or for workloads requiring BF16 precision. At $0.10–$0.35/hr on GCP, AWS, and other providers, it remains the cheapest data center GPU for inference despite its age.