Batch AI Processing Cost Guide: APIs, Spot GPUs, and Pipelines (2026)

Batch processing — running AI inference jobs asynchronously over large datasets rather than serving real-time requests — is the most cost-effective way to use LLMs in 2026. Managed API providers offer dedicated batch endpoints at 50% of standard pricing. Self-hosted GPU spot instances can cut costs a further 40–70% vs on-demand. Combined, a well-architected batch pipeline can process the same workload at 10–20× lower cost than synchronous real-time inference.

The trade-off is latency: batch jobs accept turnaround times of minutes to hours rather than milliseconds. This makes batch processing ideal for document processing, data enrichment, content generation pipelines, evaluation runs, embedding generation, and any workload where results aren't needed in real time. This guide covers batch API pricing, self-hosted batch architectures, GPU selection, queue design, and the cost math at every scale from 100K to 1B tokens/day. Use the LLM ROI calculator to model your specific batch workload, and live GPU prices to compare current spot rates.

All three major frontier providers offer dedicated batch endpoints at a 50% discount to standard pricing. Jobs are queued and processed within a defined SLA window (typically 24 hours) rather than served in real time.

OpenAI Batch API: GPT-4o: $1.25/1M input, $5.00/1M output (50% off standard). GPT-4o Mini: $0.075/1M input, $0.30/1M output (50% off). Maximum batch size: 50,000 requests or 200MB per batch file. Completion window: 24 hours. Supports all standard Chat Completions parameters including function calling and structured outputs.

Anthropic Message Batches API: Claude 3.5 Sonnet: $1.50/1M input, $7.50/1M output (50% off). Claude 3.5 Haiku: $0.40/1M input, $2.00/1M output (50% off). Maximum batch size: 10,000 requests. Completion window: 24 hours. Results available via polling or webhook.

Google Batch Prediction (Vertex AI): Gemini 1.5 Pro: $0.3125/1M input, $1.25/1M output (75% off standard — best batch discount). Gemini 1.5 Flash: $0.01875/1M input, $0.075/1M output. Completion window: 6–48 hours depending on job size. Requires Vertex AI project setup.

At 100M tokens/day (large batch workload): OpenAI GPT-4o Mini batch: $3,750/month. Anthropic Claude 3.5 Haiku batch: $7,500/month. Google Gemini 1.5 Flash batch: $562/month. DeepSeek V3 (no dedicated batch endpoint but off-peak pricing): $0.135/1M input = $405/month. Gemini 1.5 Flash batch is the cheapest frontier-quality batch option by a wide margin.

For workloads above ~500M tokens/month, self-hosted batch processing on spot GPUs beats managed batch APIs on cost. The key is designing for interruption: spot instances can be reclaimed with 30–90 seconds notice, so batch jobs must checkpoint progress and resume cleanly.

Spot GPU pricing vs on-demand (2026): RTX 4090 on-demand: $0.44/hr. Spot on Vast.ai: $0.12–$0.18/hr (60–73% saving). Spot on RunPod: $0.18–$0.24/hr (45–59% saving). A100 80GB on-demand: $1.89/hr (Lambda Labs). Spot on Vast.ai: $0.60–$0.90/hr (52–68% saving). H100 80GB on-demand: $2.49/hr. Spot on CoreWeave: $1.20–$1.60/hr (36–52% saving).

Throughput for batch inference (vLLM with continuous batching, 70% GPU utilization): Llama 3.1 8B on RTX 4090 spot ($0.15/hr): ~3,500 tokens/sec → $0.012/1M tokens. Llama 3.1 70B on A100 80GB spot ($0.75/hr): ~1,400 tokens/sec → $0.149/1M tokens. Llama 3.1 70B on H100 spot ($1.40/hr): ~2,200 tokens/sec → $0.177/1M tokens.

Comparing self-hosted spot to managed batch: Llama 3.1 8B spot vs GPT-4o Mini batch ($0.1875/1M blended): Self-hosted wins by 15× on cost. Quality gap is real for complex tasks but negligible for classification, extraction, and summarization. Llama 3.1 70B spot vs Claude 3.5 Haiku batch ($1.20/1M blended): Self-hosted wins by 8× on cost with comparable quality for most batch tasks.

A production batch processing system has four components: job ingestion, queue management, worker pool, and result storage. Each has cost implications.

Job ingestion: Accept batch jobs via API or file upload. Store raw inputs in object storage (S3-compatible): $0.023/GB/month. A 1M-document batch at 2KB/document = 2GB = $0.046/month storage. Negligible.

Queue management: Redis or SQS for job queuing. Redis on a small VM ($0.05/hr): $36/month. AWS SQS: $0.40/1M requests — at 1M jobs/day: $12/month. Use dead-letter queues for failed jobs and priority queues to separate urgent from background work.

Worker pool (the main cost): Spot GPU instances running vLLM or TGI. Scale workers based on queue depth. Key design decisions: (1) Minimum workers: keep 1 warm instance to avoid cold-start delays on new jobs. (2) Maximum workers: cap based on budget, not just queue depth. (3) Checkpointing: save progress every 1,000 items so spot interruptions lose at most 1,000 items of work. (4) Retry logic: automatic retry with exponential backoff for interrupted jobs.

Result storage: Write completed results to object storage. At 1KB/result × 1M results/day = 1GB/day = 30GB/month = $0.69/month. For large outputs (long-form generation), budget 5–10× more.

Total infrastructure overhead beyond GPU compute: ~$50–$100/month for queue, storage, and coordination. Amortized over large batch volumes, this is under 1% of total cost.

Different batch workloads have different token profiles and optimal model choices. Here are the key workload types with cost benchmarks:

Document classification and extraction (short input, short output): Typical: 500 input + 50 output tokens. Best model: fine-tuned Llama 3.1 8B or GPT-4o Mini batch. Cost with GPT-4o Mini batch: $0.0000469/document. Cost with self-hosted Llama 3.1 8B spot: $0.0000066/document. At 1M documents/day: GPT-4o Mini batch = $1,407/month. Self-hosted 8B spot = $198/month.

Content generation (medium input, long output): Typical: 1,000 input + 1,000 output tokens. Best model: Claude 3.5 Haiku batch or Gemini 1.5 Flash batch. Cost with Gemini 1.5 Flash batch: $0.0000938/document. Cost with self-hosted Llama 3.1 70B spot: $0.000298/document. Gemini 1.5 Flash batch wins on cost here — managed API beats self-hosted for long-output generation tasks.

Data enrichment / entity extraction (long input, short output): Typical: 4,000 input + 200 output tokens. Best model: GPT-4o Mini batch or Gemini 1.5 Flash batch. Cost with GPT-4o Mini batch: $0.000360/document. Cost with self-hosted Llama 3.1 8B spot: $0.0000528/document. Self-hosted wins by 7× for long-context extraction.

Embedding generation (no output, just vectors): OpenAI text-embedding-3-small batch: $0.010/1M tokens (50% off standard). Self-hosted BGE-M3 on RTX 4090 spot ($0.15/hr): ~$0.00082/1M tokens. Self-hosted wins by 12× above ~200M tokens/month.

LLM evaluation runs (benchmark your model): Typical: 2,000 input + 500 output tokens per eval sample. Use cheapest capable model — Gemini 1.5 Flash batch at $0.000069/sample. 10,000-sample eval suite: $0.69. Run evals freely — the cost is negligible.

Single-GPU batch throughput is often the bottleneck for large-scale pipelines. Here's how to scale horizontally and the cost implications.

Single-GPU throughput ceiling: RTX 4090 running Llama 3.1 8B: ~3,500 tokens/sec = 302M tokens/day. At 500 tokens/document: 604,800 documents/day. Sufficient for most batch workloads under 500K documents/day.

Multi-GPU scaling (linear throughput, linear cost): 4× RTX 4090 spot ($0.60/hr total on Vast.ai): ~14,000 tokens/sec = 1.2B tokens/day = 2.4M documents/day. Monthly cost: $432. Cost per document (500 tokens): $0.0000060. 8× A100 80GB spot ($6.00/hr total): ~11,200 tokens/sec for Llama 3.1 70B = 967M tokens/day. Monthly cost: $4,320. Cost per document (500 tokens): $0.0000134.

Throughput vs latency trade-off: For batch jobs, maximize throughput over latency. Use the largest batch size that fits in VRAM (vLLM handles this automatically with continuous batching). Increase max_tokens conservatively — unused token budget still consumes KV cache memory. Use tensor parallelism across GPUs only for 70B+ models; for 8B, independent single-GPU workers are more efficient.

Autoscaling strategy: Use queue depth as the scaling signal. Scale up when queue depth > 10,000 items; scale down when queue depth < 1,000 items. Add a 5-minute cooldown to avoid thrashing. With spot instances, pre-warm replacement workers before terminating existing ones to maintain throughput during scale-in events. Compare GPU compute pricing to find the best spot availability for your target GPU.

The right choice depends on volume, model quality requirements, and engineering capacity.

Use managed batch APIs when: Volume is under 500M tokens/month (OpenAI/Anthropic batch APIs are cost-competitive). You need frontier model quality (GPT-4o, Claude 3.5 Sonnet) — no self-hosted equivalent. You have no DevOps capacity for spot instance management. You need guaranteed 24-hour SLA with no infrastructure risk. Workload is bursty and unpredictable — managed APIs absorb spikes without pre-provisioning.

Use self-hosted spot GPUs when: Volume exceeds 500M tokens/month for 8B-equivalent tasks. You can tolerate occasional job delays from spot interruptions (with proper checkpointing). You need custom fine-tuned models not available via managed APIs. You process sensitive data that cannot leave your infrastructure. You're running continuous background pipelines (embeddings, classification, enrichment) where cost is the primary constraint.

Hybrid architecture (recommended for most scaled products): Use managed batch APIs (OpenAI, Anthropic, Google) for: frontier-quality tasks, burst capacity, and jobs with strict 24-hour SLAs. Use self-hosted spot GPUs for: high-volume commodity tasks (classification, extraction, embedding), fine-tuned model inference, and always-on background pipelines. Route jobs to the appropriate tier based on quality requirements and urgency. This hybrid approach typically reduces total batch processing cost by 50–70% vs managed-only while maintaining quality where it matters.

A complete TCO breakdown for a production batch pipeline processing 10M documents/month (5B tokens/month at 500 tokens/document).

Option A — Managed batch APIs only (GPT-4o Mini batch): Input tokens (400/doc × 10M): 4B × $0.075/1M = $300/month. Output tokens (100/doc × 10M): 1B × $0.30/1M = $300/month. Total: $600/month. Cost per document: $0.000060. Zero ops overhead.

Option B — Self-hosted Llama 3.1 8B on spot RTX 4090 (Vast.ai, $0.15/hr): Required throughput: 10M docs × 500 tokens ÷ (30 days × 86,400 sec) = 1,929 tokens/sec. GPUs needed: 1× RTX 4090 (3,500 tokens/sec capacity with headroom). Monthly GPU cost: $0.15/hr × 720hr = $108. Infrastructure overhead (queue, storage, monitoring): $50/month. Total: $158/month. Cost per document: $0.0000158. 74% cheaper than Option A.

Option C — Hybrid (managed for quality tasks, self-hosted for commodity): 20% of documents need GPT-4o Mini quality → 2M docs via managed batch: $120/month. 80% are classification/extraction → 8M docs via self-hosted 8B spot: $126/month. Total: $246/month. Cost per document: $0.0000246. 59% cheaper than managed-only with quality preserved where needed.

At 100M documents/month (50B tokens), Option B scales to ~$1,080/month (10× RTX 4090 spot) vs $6,000/month for managed batch — a 82% saving. Monitor GPU spot prices to track availability and optimize provider selection across Vast.ai, RunPod, and Lambda Labs.