LLM API vs Self-Hosted: The Definitive Cost Comparison for 2026

The decision between managed LLM APIs and self-hosted open-weight models is the highest-leverage cost decision in any AI product. Get it right and you can cut inference spend by 70–95% at scale. Get it wrong and you either overpay for managed APIs or sink engineering time into infrastructure that doesn't pay off at your volume.

The trade-off is simple: managed APIs (OpenAI, Anthropic, Google) charge a per-token markup of 5–50× over raw GPU compute cost, but include zero ops overhead, automatic scaling, and access to frontier models that can't be self-hosted. Self-hosted open-weight models (Llama, Mistral, Qwen, DeepSeek) eliminate the markup but require GPU infrastructure, model serving, and engineering time to maintain.

This guide gives you the exact numbers to make the call: cost-per-token comparisons, break-even volumes by model tier, hidden costs on both sides, and a decision framework for every scale from prototype to 100M+ daily tokens. Use the LLM ROI calculator to model your specific workload, and live GPU prices to compare current self-hosting rates.

Managed API pricing in 2026 spans a 200× range from cheapest to most expensive. Understanding the tiers is essential before comparing to self-hosted.

Frontier tier (highest quality, highest cost): OpenAI GPT-4o: $2.50/1M input, $10.00/1M output. Anthropic Claude 3.5 Sonnet: $3.00/1M input, $15.00/1M output. Google Gemini 1.5 Pro: $1.25/1M input, $5.00/1M output. xAI Grok-2: $2.00/1M input, $10.00/1M output.

Advanced tier (strong quality, moderate cost): GPT-4o Mini: $0.15/1M input, $0.60/1M output. Claude 3.5 Haiku: $0.80/1M input, $4.00/1M output. Gemini 2.0 Flash: $0.10/1M input, $0.40/1M output. Mistral Mistral Small: $0.10/1M input, $0.30/1M output.

Efficient tier (good quality, low cost): Groq Llama 3.3 70B: $0.59/1M input, $0.79/1M output. Together AI Llama 3.1 8B: $0.06/1M input, $0.06/1M output. Fireworks AI Llama 3.1 8B: $0.10/1M input, $0.10/1M output. Cerebras Llama 3.3 70B: $0.60/1M input, $0.60/1M output. DeepSeek V3: $0.27/1M input, $1.10/1M output.

At 1M tokens/day (a mid-scale product): GPT-4o: $3,750–$12,500/month (depending on input/output ratio). GPT-4o Mini: $56–$225/month. Together AI Llama 3.1 8B: $1.80–$1.80/month. The 2,000× cost gap between frontier and efficient tiers is the most important number in LLM economics.

Self-hosted inference has three cost components: GPU compute, model serving infrastructure, and engineering time. The first is easy to calculate; the latter two are often underestimated.

GPU compute for common model sizes (on-demand rates, 2026): 7B–8B models (Llama 3.1 8B, Mistral 7B): Fit on single RTX 4090 (24GB VRAM). $0.35–$0.50/hr. Throughput: ~2,000–4,000 tokens/second with vLLM continuous batching. 13B–14B models: Fit on RTX 6000 Ada (48GB) or 2× RTX 4090. $0.80–$1.00/hr. 70B models (Llama 3.3 70B, Qwen 2.5 72B): Require 2× A100 80GB or 1× H100 80GB. $2.49–$4.98/hr. Throughput: ~800–1,500 tokens/second. 405B models (Llama 3.1 405B): Require 8–10× H100 80GB. $20–$25/hr. Throughput: ~300–600 tokens/second.

Model serving overhead: vLLM, TGI, or TensorRT-LLM add ~$50–$150/month per node for load balancer, monitoring, and autoscaling. Budget 10–15% on top of raw GPU cost.

Engineering time: The largest hidden cost. Initial setup: 8–20 hours. Ongoing maintenance (updates, incident response, capacity planning): 4–8 hours/month. At $150/hr fully-loaded engineering cost: $600–$1,200/month ongoing. This alone exceeds the managed API cost for many low-to-mid volume workloads. Compare GPU compute pricing to find the best rates for your target GPU.

Direct cost-per-token comparison at production scale. Self-hosted costs assume always-on GPU, vLLM serving, and 50% average GPU utilization (realistic for production with variable traffic).

8B model tier: Together AI Llama 3.1 8B (managed): $0.060/1M tokens. Self-hosted Llama 3.1 8B on RTX 4090 ($0.44/hr, 3,000 tokens/sec at 50% util): $0.081/1M tokens. Managed API wins at this tier — the markup is only 35% and eliminates all ops overhead.

70B model tier: Groq Llama 3.3 70B (managed): $0.69/1M tokens blended. Self-hosted Llama 3.3 70B on H100 80GB ($2.49/hr, 1,200 tokens/sec at 50% util): $0.575/1M tokens. Self-hosted wins by ~17% on raw compute — but only after accounting for engineering overhead does the real break-even emerge.

Frontier tier (GPT-4o equivalent quality): GPT-4o (managed): $6.25/1M tokens blended. No self-hosted equivalent — GPT-4o architecture is closed. Nearest open-weight: Llama 3.1 405B self-hosted on 8× H100 ($19.92/hr, 450 tokens/sec at 50% util): $2.46/1M tokens. 60% cheaper than GPT-4o — but requires $15,000+/month in GPU spend to justify the fixed cost.

Key insight: for 8B models, managed APIs are cost-competitive or cheaper than self-hosting at almost any volume due to the low markup. For 70B models, self-hosting wins above ~50M tokens/month. For frontier-quality tasks, GPT-4o has no cost-competitive self-hosted alternative until very high volumes.

The break-even point — where self-hosted becomes cheaper than managed API — varies dramatically by model tier and GPU choice.

8B model break-even (RTX 4090 at $0.44/hr vs Together AI at $0.06/1M tokens): Self-hosted monthly fixed cost: $317 GPU + $50 overhead = $367/month. Together AI variable cost: $0.06/1M tokens. Break-even: $367 ÷ ($0.081 - $0.060)/1M tokens = 17.5B tokens/month = 583M tokens/day. Self-hosting an 8B model almost never beats Together AI on pure cost — the managed API markup is too small. Self-hosting 8B is justified for data privacy, custom fine-tunes, or latency requirements, not cost.

70B model break-even (H100 at $2.49/hr vs Groq at $0.69/1M blended): Self-hosted monthly fixed cost: $1,800 GPU + $150 overhead = $1,950/month. Self-hosted variable cost: $0.575/1M tokens. Groq variable cost: $0.690/1M tokens. Savings per token: $0.115/1M. Break-even: $1,950 ÷ $0.115/1M = 16.96B tokens/month = 565M tokens/day. At 1B tokens/day (a large-scale product), self-hosted 70B saves $3,450/month over Groq.

Frontier-quality break-even (8× H100 at $19.92/hr vs GPT-4o at $6.25/1M blended): Self-hosted monthly fixed: $14,342 GPU + $500 overhead = $14,842/month. Savings per token vs GPT-4o: $3.79/1M. Break-even: $14,842 ÷ $3.79/1M = 3.92B tokens/month = 130M tokens/day. Above 130M tokens/day of frontier-quality inference, self-hosted Llama 3.1 405B beats GPT-4o on cost. Track GPU spot prices — spot H100 clusters can cut the fixed cost by 40–60%.

Both managed APIs and self-hosted have significant hidden costs that don't appear in the headline per-token or per-hour rates.

Hidden costs of managed APIs: Rate limits and throttling — frontier APIs impose RPM/TPM limits that require retry logic and can cause latency spikes at scale. Vendor lock-in — migrating from OpenAI to Anthropic requires prompt re-engineering (system prompts, tool call formats differ). Context window pricing — long-context requests (100K+ tokens) cost proportionally more; a single 100K-token request to GPT-4o costs $0.25 in input tokens alone. Egress costs — API responses count as output tokens billed at 3–5× input rates. Prompt caching limitations — OpenAI caches automatically but only for exact prefix matches; Anthropic requires explicit cache control headers.

Hidden costs of self-hosted: Cold start latency — loading a 70B model from NVMe takes 30–90 seconds; always-on instances eliminate this but add fixed cost. VRAM fragmentation — at high concurrency, KV cache fills VRAM and degrades throughput; requires careful vLLM configuration. Model updates — open-weight model releases (Llama 4, Qwen 3, etc.) require re-evaluation, re-testing, and re-deployment. Quantization trade-offs — INT4/INT8 quantization reduces VRAM requirements by 2–4× but degrades quality 2–8% on benchmarks; requires validation for your use case. Multi-GPU networking — 70B+ models on multi-GPU nodes require NVLink or fast interconnect; H100 SXM5 with NVLink is significantly faster than PCIe for multi-GPU inference.

Cost is only half the equation. Managed frontier APIs offer quality advantages that are genuinely hard to replicate with open-weight models in 2026.

Tasks where frontier APIs justify the premium: Complex multi-step reasoning (math, code, legal analysis) — GPT-4o and Claude 3.5 Sonnet score 15–25% higher than Llama 3.1 70B on MMLU-Pro and MATH benchmarks. Instruction following precision — frontier models follow complex, nuanced instructions more reliably; critical for structured output, tool use, and agentic workflows. Long-context coherence — Google Gemini 1.5 Pro with 1M context window has no open-weight equivalent; Anthropic Claude with 200K context outperforms open models at long-document tasks. Vision and multimodal — GPT-4o Vision and Claude 3.5 Sonnet vision capabilities exceed open-weight alternatives for complex image understanding.

Tasks where open-weight models match frontier quality: RAG-based Q&A — retrieval provides the knowledge; generation quality gap narrows significantly. Classification and extraction — Llama 3.1 8B with few-shot prompting matches GPT-4o Mini on most classification tasks. Code generation (common languages) — DeepSeek Coder V2 and Qwen 2.5 Coder match GPT-4o on Python/JavaScript benchmarks. Summarization — 70B models match frontier quality for most summarization tasks. Chat and customer service — fine-tuned 8B–13B models match frontier quality for domain-specific chat with proper training data.

The practical rule: use frontier APIs for tasks where quality directly drives revenue (customer-facing, high-stakes decisions). Use efficient managed APIs or self-hosted for internal tools, batch processing, and tasks with clear quality ceilings.

A practical decision tree based on daily token volume, quality requirements, and ops capacity.

Under 100M tokens/month (prototype to small product): Use managed APIs exclusively. Google Gemini 2.0 Flash or Mistral Mistral Small for cost-sensitive tasks; GPT-4o Mini for quality-sensitive tasks. Zero ops overhead. Total LLM cost: $10–$300/month. Self-hosting is not justified at this scale.

100M–1B tokens/month (growing product): Managed APIs still win for 8B-equivalent tasks (Together AI, Fireworks AI). Evaluate self-hosting for 70B tasks if you have DevOps capacity — break-even is ~565M tokens/day for 70B. Implement prompt caching on Anthropic and OpenAI to reduce costs 40–80% on stable prompts. Total LLM cost: $300–$3,000/month.

1B–10B tokens/month (scaled platform): Self-host 70B models on H100 for high-volume tasks. Keep managed frontier APIs for complex reasoning tasks. Use Groq or Cerebras for latency-sensitive paths. Total LLM cost: $3,000–$30,000/month.

Above 10B tokens/month (enterprise scale): Full self-hosted stack for commodity tasks (8B, 70B). Managed frontier APIs only for tasks requiring GPT-4o/Claude quality. Consider fine-tuning smaller models on your domain data — a fine-tuned 8B can match 70B quality for narrow tasks at 10× lower cost. Compare H100 cluster pricing across Lambda Labs, CoreWeave, and Nebius AI for the best enterprise rates.