RAG Pipeline Cost Guide: Embeddings, Vector DB, and LLM Inference (2026)

Retrieval-Augmented Generation (RAG) pipelines have four distinct cost layers: (1) embedding model inference to convert documents and queries into vectors, (2) vector database hosting to store and search those embeddings, (3) retrieval compute to run similarity search at query time, and (4) LLM inference to generate the final answer from retrieved context. Each layer has its own cost structure and optimization levers. Getting all four right is the difference between a $0.001/query pipeline and a $0.05/query pipeline — a 50× cost gap at the same quality level.

This guide breaks down each layer with real 2026 pricing, worked examples at 10K–10M daily queries, and the key decisions that drive cost. Use the LLM ROI calculator to model the inference layer, and live GPU prices to compare self-hosted embedding options.

Embedding models convert text chunks and queries into dense vectors. Every document ingested and every query at runtime requires an embedding call. Costs compound quickly at scale.

Managed embedding APIs (2026 rates): OpenAI text-embedding-3-small: $0.020/1M tokens. OpenAI text-embedding-3-large: $0.130/1M tokens. Cohere embed-v3: $0.100/1M tokens. Google text-embedding-004: $0.000/1M tokens (free up to 1M/day, then $0.025/1M). Mistral embed: $0.100/1M tokens.

At a typical 500-token chunk size, embedding 1 million documents costs: OpenAI small: $10. OpenAI large: $65. Cohere: $50. This is a one-time ingestion cost — re-embedding is only needed when documents change or you switch models.

Query-time embedding is the ongoing cost. At 500 tokens/query: OpenAI small: $0.00001/query. At 100K queries/day: $1.00/day = $30/month. At 1M queries/day: $300/month just for query embeddings.

Self-hosted embedding with BGE-M3 or E5-large on RTX 4090 ($0.44/hr): ~50,000 tokens/second throughput. At 500 tokens/query: 100 queries/second = 8.6M queries/day capacity. Monthly cost: $317 (always-on). Break-even vs OpenAI small: ~1.06M queries/day. For most RAG deployments under 1M queries/day, managed embedding APIs are cheaper than self-hosting.

The vector database stores embeddings and runs similarity search at query time. Cost structure varies significantly by provider — some charge by storage, some by compute, some by both.

Managed vector databases (2026 pricing): Pinecone Serverless: $0.033/1M read units + $0.08/GB storage/month. At 100K queries/day with top-10 retrieval: ~$100/month for reads + storage. Pinecone Pod-based: $0.096/hr per p1 pod (1M vectors). Weaviate Cloud: $0.095/hr for standard cluster (25M vectors). Qdrant Cloud: $0.014/hr for 1GB RAM cluster. Chroma Cloud: $0.025/1M queries. Zilliz (managed Milvus): $0.10/hr for 1-node cluster.

Self-hosted vector databases: Qdrant, Milvus, Weaviate, and pgvector (PostgreSQL extension) are all open-source and free to self-host. Infrastructure cost only: a 4-core/16GB RAM VM at $0.10–$0.20/hr handles ~10M vectors with sub-10ms search latency. Monthly cost: $72–$144/month for a dedicated vector DB node.

pgvector on existing PostgreSQL: If you already run PostgreSQL, pgvector adds vector search with zero additional infrastructure cost. Performance is 2–5× slower than dedicated vector DBs at scale (>1M vectors), but adequate for most RAG deployments under 500K vectors.

For 10M+ vectors with <5ms p99 latency requirements, self-hosted Qdrant or Milvus on a memory-optimized VM ($0.30–$0.60/hr for 64GB RAM) is the most cost-effective option. Compare GPU compute pricing — many offer CPU/memory-optimized instances alongside GPU nodes.

Retrieval compute is the cost of running the similarity search itself — finding the top-K most relevant chunks for each query. For managed vector databases, this is included in the per-query or per-read-unit pricing. For self-hosted, it's part of your VM/GPU cost.

Approximate retrieval latency benchmarks (10M vectors, top-10 results): Pinecone Serverless: 20–80ms p99. Qdrant self-hosted (CPU, 32GB RAM): 5–15ms p99. Weaviate Cloud: 15–50ms p99. pgvector (IVFFlat index): 50–200ms p99. Milvus self-hosted (GPU index on RTX 4090): 1–5ms p99.

GPU-accelerated vector search: For latency-critical RAG (sub-10ms end-to-end retrieval), GPU-accelerated FAISS or Milvus with GPU index on RTX 4090 achieves 1–5ms p99 at 10M vectors. Cost: $0.44/hr = $317/month. Justified only for >100K queries/day where retrieval latency is a bottleneck.

Hybrid search (dense + sparse): Combining vector similarity with BM25 keyword search improves retrieval quality significantly for most RAG workloads. Adds ~20–40% compute overhead but reduces the number of LLM calls needed (higher first-pass precision means fewer re-ranking steps). Most managed vector DBs support hybrid search natively — check Cohere Rerank API ($2.00/1M tokens) for cross-encoder re-ranking if precision is critical.

LLM inference is typically the largest cost layer in a RAG pipeline. RAG prompts are inherently long — retrieved context chunks (500–2,000 tokens each × top-K results) plus the user query plus system prompt. A typical RAG prompt: 3,000–8,000 input tokens, 200–500 output tokens.

Cost at 5,000 input + 300 output tokens per query: OpenAI GPT-4o: $0.01550. GPT-4o Mini: $0.000930. Anthropic Claude 3.5 Haiku: $0.001340. Google Gemini 2.0 Flash: $0.000620. Groq Llama 3.3 70B: $0.003190. Together AI Llama 3.1 8B: $0.000360. Mistral Mistral Small: $0.000690.

At 100K queries/day: GPT-4o: $1,550/day = $46,500/month. GPT-4o Mini: $93/day = $2,790/month. Gemini 2.0 Flash: $62/day = $1,860/month. Together AI Llama 3.1 8B: $36/day = $1,080/month.

Prompt caching is the single biggest lever for RAG cost reduction. If your system prompt and retrieval context are partially stable (e.g., same document corpus), Anthropic prompt caching saves 90% on cached input tokens, OpenAI saves 50%. For a RAG pipeline where 3,000 of 5,000 input tokens are cacheable: Claude 3.5 Haiku with caching: $0.000214/query vs $0.001340 without — 84% reduction. See the LLM ROI calculator to model caching savings for your workload.

Putting all four layers together — embedding, vector DB, retrieval, and LLM inference — here are full pipeline costs at three scales. Assumes: 500-token chunks, top-5 retrieval, 5,000 input + 300 output tokens to LLM, GPT-4o Mini for generation.

10,000 queries/day (small production app): Embeddings (OpenAI small): $0.10/day. Vector DB (Pinecone Serverless): $3.30/day. LLM inference (GPT-4o Mini): $9.30/day. Total: ~$12.70/day = $381/month. Cost per query: $0.00127.

100,000 queries/day (mid-scale product): Embeddings (OpenAI small): $1.00/day. Vector DB (Pinecone Serverless): $33/day. LLM inference (GPT-4o Mini): $93/day. Total: ~$127/day = $3,810/month. Cost per query: $0.00127.

1,000,000 queries/day (high-scale platform): Embeddings (self-hosted BGE-M3 on RTX 4090): $10.57/day ($317/month). Vector DB (self-hosted Qdrant, 64GB RAM VM): $4.80/day ($144/month). LLM inference (Gemini 2.0 Flash or Together AI Llama 3.1 8B): $360–$620/day. Total: ~$375–$635/day = $11,250–$19,050/month. Cost per query: $0.000375–$0.000635.

Switching from GPT-4o Mini to Gemini 2.0 Flash or Together AI Llama 3.1 8B at 1M queries/day saves $8,000–$16,000/month with comparable quality for most RAG use cases.

Seven high-impact optimizations that reduce RAG pipeline costs by 40–90%:

1. Reduce retrieved context size: Retrieve top-3 instead of top-10, use smaller chunks (256 tokens vs 512), and implement re-ranking to improve precision before passing to LLM. Cutting input tokens from 5,000 to 2,500 halves LLM inference cost.

2. Use prompt caching: Anthropic Claude caches up to 90% of stable input tokens at 10% of normal cost. OpenAI automatic caching saves 50% on repeated prefixes. For RAG with a fixed document corpus, caching the retrieved context prefix can save 60–80% on input token costs.

3. Downgrade the LLM for simple queries: Route simple factual queries to Together AI Llama 3.1 8B ($0.000360/query) and complex reasoning to GPT-4o Mini ($0.000930/query). A 70/30 split saves ~25% on LLM costs with minimal quality impact.

4. Self-host embeddings at scale: Above 500K queries/day, self-hosted BGE-M3 on RTX 4090 ($317/month) beats OpenAI text-embedding-3-small ($3,000/month). Break-even: ~1M queries/day.

5. Cache query embeddings: Many RAG queries are repeated (same question, different users). Cache embeddings for the top 10% of queries — typically covers 40–60% of traffic. Reduces embedding API calls by 40–60%.

6. Batch document ingestion: Embed documents in large batches (1,000+ chunks per API call) to maximize throughput and minimize per-call overhead. OpenAI embedding API supports up to 2,048 inputs per request.

7. Use Groq for latency-sensitive RAG: Groq's LPU delivers 500–800 tokens/second output — 10–20× faster than standard GPU inference. For interactive RAG where time-to-first-token matters, Groq Llama 3.3 70B at $0.003190/query often beats GPT-4o Mini on both cost and latency.

Stack recommendations by query volume and budget:

Under 10K queries/day (prototype / early product): Managed everything. OpenAI text-embedding-3-small + Pinecone Serverless + GPT-4o Mini. Zero ops overhead. Total: ~$40–$130/month. Focus on retrieval quality, not cost optimization.

10K–100K queries/day (growing product): Managed embeddings + self-hosted vector DB. OpenAI small embeddings + self-hosted Qdrant ($72/month) + Google Gemini 2.0 Flash or Mistral Mistral Small for generation. Total: $500–$2,000/month. Implement prompt caching and query embedding cache.

100K–1M queries/day (scaled platform): Self-hosted embeddings on RTX 4090 + self-hosted Qdrant on memory-optimized VM + Together AI or Groq for generation. Total: $3,000–$12,000/month. Implement query routing (simple vs complex), hybrid search, and re-ranking.

Above 1M queries/day (enterprise scale): Full self-hosted stack. BGE-M3 embeddings on A100 80GB cluster + Milvus with GPU index + self-hosted Llama 3.1 70B on H100 cluster. Total: $15,000–$50,000/month depending on SLA. Compare GPU compute pricing for H100 cluster rates — Lambda Labs, CoreWeave, and Nebius AI offer the most competitive H100 pricing.