Compute Comparison
Cost Guides11 min read

Real-Time AI Inference Cost Guide: Self-Hosted vs Managed API (2026)

Cost breakdown and ROI analysis for real-time AI inference — self-hosted GPU vs managed LLM API, latency budgets, autoscaling strategies, and cost-per-request benchmarks.

Real-Time AI Inference: What It Actually Costs

Real-time AI inference — serving model predictions with sub-second or low-second latency — is the most cost-sensitive AI compute workload. Unlike training, which runs once, inference runs continuously at production scale. A model serving 10,000 requests per day at $0.002/request costs $7,300/year; at 1M requests/day, that's $730,000/year. Getting the cost-per-request right is the difference between a profitable product and one that bleeds money.

In 2026, you have two fundamentally different options: self-hosted GPU inference (rent GPU cloud compute, run your own model server) or managed LLM API inference (pay per token to OpenAI, Anthropic, Google, Groq, etc.). This guide breaks down the real costs of both approaches, with worked examples for common model sizes and request volumes. Use the live GPU price comparison and live LLM API prices to plug in current rates.

Self-Hosted GPU Inference: Cost Components

Self-hosted inference has four cost components: (1) GPU compute, (2) model serving overhead, (3) networking/egress, and (4) engineering time.

GPU compute is the dominant cost. A single H100 80GB at $2.49/hr (Lambda Labs on-demand) costs $1,800/month. An A10G at $0.75/hr costs $540/month. An RTX 4090 at $0.44/hr costs $317/month. The right GPU depends on your model size — a 7B parameter model in FP16 requires ~14GB VRAM (fits on RTX 4090), a 70B model requires ~140GB VRAM (requires 2× H100 80GB), a 405B model requires ~810GB VRAM (requires 10–11× H100 80GB).

Model serving overhead adds 10–20% to effective compute cost: inference server (vLLM, TGI, TensorRT-LLM), load balancer, monitoring, and autoscaling infrastructure. Networking egress at $0.08–$0.09/GB adds up for large response payloads. Engineering time to maintain the stack is often the largest hidden cost — budget 0.5–1 FTE for a production inference cluster. Compare GPU compute pricing to find the best on-demand rates for your target GPU.

Managed LLM API Inference: Cost Components

Managed API inference charges per token — input tokens (your prompt) and output tokens (the model's response) are billed separately, with output tokens typically 3–5× more expensive than input tokens.

Representative 2026 rates for frontier models: GPT-4o (OpenAI): $2.50/1M input, $10.00/1M output. Claude 3.5 Sonnet (Anthropic): $3.00/1M input, $15.00/1M output. Gemini 1.5 Pro (Google): $1.25/1M input, $5.00/1M output. For efficient models: GPT-4o Mini: $0.15/1M input, $0.60/1M output. Gemini 2.0 Flash: $0.10/1M input, $0.40/1M output. Groq Llama 3.1 70B: $0.59/1M input, $0.79/1M output.

A typical chatbot request with 500 input tokens and 200 output tokens costs: GPT-4o: $0.00325/request. GPT-4o Mini: $0.000195/request. Groq Llama 3.1 70B: $0.000453/request. At 100,000 daily requests, that's $325/day for GPT-4o vs $19.50/day for GPT-4o Mini. See the LLM ROI calculator to model your specific workload across 262+ models.

Break-Even Analysis: Self-Hosted vs Managed API

The break-even point between self-hosted and managed API depends on request volume, model choice, and latency requirements. Here's the math for a 7B model (Llama 3.1 8B) vs GPT-4o Mini:

Self-hosted 7B on RTX 4090 ($0.44/hr): Throughput ~500 tokens/second at batch size 8. At 500 input + 200 output tokens per request: ~0.7 requests/second = 60,000 requests/day. Monthly cost: $317 GPU + $50 overhead = $367/month. Cost per request: $0.000204.

GPT-4o Mini (OpenAI) at same request profile: $0.000195/request. Monthly cost at 60,000 req/day: $351/month.

At 60,000 requests/day, GPT-4o Mini and self-hosted 7B are nearly cost-equivalent — but self-hosted requires engineering overhead. The break-even shifts in favor of self-hosting at higher volumes: at 200,000 requests/day, self-hosted requires 3–4 RTX 4090s ($1,100/month) vs GPT-4o Mini at $1,170/month. Above ~150,000 requests/day for a 7B model, self-hosting typically wins on cost. Track GPU spot prices to time infrastructure purchases.

Latency Budgets and Their Cost Implications

Latency requirements fundamentally change the cost equation. Real-time inference typically targets one of three latency tiers:

Sub-100ms (autocomplete, search, real-time suggestions): Requires dedicated GPU capacity — no cold starts, no queuing. Self-hosted with reserved instances or managed APIs with low-latency endpoints (Groq, Cerebras). Groq LPU delivers 500–800 tokens/second output speed — the fastest available for Llama models. Cost premium: 2–5× vs standard API rates.

Sub-2s (chatbots, copilots, interactive tools): Standard managed API or self-hosted with moderate autoscaling. Most frontier models achieve this at normal load. Cost: standard rates apply.

Sub-10s (document analysis, code generation, complex reasoning): Batch-friendly, can use spot instances and queue-based architectures. Significant cost savings possible — 40–60% vs on-demand. See the batch processing cost guide for async workload optimization.

For sub-100ms requirements, Groq and Cerebras are the only managed API options that reliably hit this target for large models. Self-hosted on H100 with TensorRT-LLM can achieve sub-100ms for 7B models at low batch sizes.

Autoscaling: The Key to Cost-Efficient Real-Time Inference

Traffic to inference endpoints is rarely flat — most production systems see 5–20× variation between peak and off-peak hours. Autoscaling is the primary lever for reducing average cost without sacrificing peak performance.

For managed APIs, autoscaling is automatic — you pay only for what you use. For self-hosted, you need to implement it explicitly. Key strategies:

Scale-to-zero for low-traffic periods: Kubernetes with KEDA (Kubernetes Event-Driven Autoscaling) can scale GPU pods to zero during off-hours. Cold start time for a 7B model on RTX 4090: ~15–30 seconds (model load from NVMe). Acceptable for non-interactive workloads; too slow for sub-2s latency targets.

Minimum replica floor: Keep 1 warm replica at all times for interactive workloads. At $0.44/hr for RTX 4090, the always-on floor costs $317/month — worth it if it eliminates cold-start latency for users.

Spot instance autoscaling: Use spot for scale-out capacity, on-demand for the minimum floor. Spot RTX 4090 from $0.15/hr (Vast.ai) vs $0.44/hr on-demand — 66% savings on burst capacity. Implement graceful shutdown handlers to drain in-flight requests before spot termination. Compare RunPod vs Vast.ai spot pricing for the best burst rates.

Cost-Per-Request Benchmarks by Model and Provider

Worked cost-per-request examples for a standard chatbot request (500 input tokens, 300 output tokens):

Frontier models (managed API): GPT-4o (OpenAI): $0.00425. Claude 3.5 Sonnet (Anthropic): $0.00600. Gemini 1.5 Pro (Google): $0.00213. Gemini 2.0 Pro: $0.00350.

Efficient models (managed API): GPT-4o Mini: $0.000255. Gemini 2.0 Flash: $0.000170. Groq Llama 3.3 70B: $0.000532. Together AI Llama 3.1 8B: $0.000060.

Self-hosted (GPU cloud, 7B model, RTX 4090 at $0.44/hr, 60K req/day): $0.000176/request. Self-hosted (70B model, 2× H100 at $2.49/hr each, 20K req/day): $0.000598/request. Self-hosted (405B model, 10× H100 SXM5 at $2.49/hr each, 5K req/day): $0.001196/request.

Key insight: for 7B models at scale, self-hosted GPU cloud is cost-competitive with the cheapest managed APIs. For 70B+ models, managed APIs like Groq and Together AI often beat self-hosted on cost until you reach very high volumes (500K+ requests/day). Use the LLM ROI calculator to model your exact workload.

Choosing the Right Inference Stack

The right inference stack depends on three variables: model size, request volume, and latency requirement.

Small models (≤7B), <50K req/day, latency >500ms: Use managed API (Together AI, Groq, Fireworks AI). Cheapest option, zero ops overhead. Together AI Llama 3.1 8B at $0.06/1M input is the lowest-cost option for this tier.

Small models (≤7B), >100K req/day: Self-hosted on RTX 4090 or A10G. Break-even vs managed API at ~80–100K req/day. Use vLLM for serving — it achieves near-optimal GPU utilization with continuous batching.

Medium models (7B–70B), any volume: Groq for sub-200ms latency; Together AI or Fireworks AI for cost-optimized; self-hosted on H100 for >200K req/day.

Large models (>70B), latency-sensitive: Managed API (OpenAI GPT-4o, Anthropic Claude, Google Gemini) for <10K req/day; self-hosted H100 cluster for higher volumes. At 405B scale, self-hosting requires significant infrastructure investment — only justified above ~1M req/day.

Monitor live LLM prices and GPU cloud rates continuously — the cost landscape shifts monthly as new models and providers enter the market.

Frequently Asked Questions

What is the cheapest way to run real-time AI inference in 2026?
For small models (≤7B parameters) at low volume (<50K req/day), managed APIs like Together AI Llama 3.1 8B ($0.06/1M tokens) or Groq are cheapest with zero ops overhead. At higher volumes (>100K req/day), self-hosted on RTX 4090 ($0.44/hr) becomes cost-competitive. For frontier models (GPT-4o, Claude), managed API is almost always cheaper than self-hosting until very high volumes.
How much does real-time LLM inference cost per request?
A standard chatbot request (500 input + 300 output tokens) costs: GPT-4o $0.00425, GPT-4o Mini $0.000255, Gemini 2.0 Flash $0.000170, Groq Llama 3.3 70B $0.000532, self-hosted 7B on RTX 4090 ~$0.000176. Costs vary 25× between the cheapest and most expensive options for equivalent quality.
When does self-hosted GPU inference beat managed API on cost?
For 7B models, self-hosting beats managed API at approximately 80–100K requests/day. For 70B models, the break-even is around 300–500K requests/day. For frontier models (GPT-4o, Claude 3.5 Sonnet), self-hosting is rarely cost-competitive due to the engineering overhead of running 10+ H100s.
What GPU is best for real-time inference?
RTX 4090 ($0.44/hr) is the best value for 7B models — 24GB VRAM fits the model with room for batching. A10G ($0.75/hr) is the best enterprise option for 13B models. H100 80GB ($2.49/hr) is required for 70B+ models and delivers the lowest latency for latency-sensitive workloads.
How do I achieve sub-100ms inference latency?
Sub-100ms requires either Groq LPU (500–800 tokens/sec output) or Cerebras for managed API, or self-hosted H100 with TensorRT-LLM for 7B models at low batch sizes. Standard GPU inference with vLLM achieves 50–200ms time-to-first-token for 7B models depending on load.
What is the cost of autoscaling GPU inference?
Keeping one warm RTX 4090 replica costs $317/month. Spot burst capacity on Vast.ai starts at $0.15/hr — 66% cheaper than on-demand. A minimum-floor + spot-burst architecture typically reduces average inference cost by 30–50% vs always-on on-demand instances.

More Guides

H100 vs A100: Which GPU Should You Use for LLM Training in 2026?The Cheapest GPU Cloud Providers for AI Inference in 2026How to Calculate LLM API Costs: A Complete Guide for 2026