RAG pipelines can get expensive fast: embedding costs, vector storage costs, LLM inference costs. After running our internal knowledge-base RAG in production for six months for 50,000 queries/month and 40 engineers, we cut total cost 64% without losing retrieval quality. Here is the breakdown, three optimizations, and the final numbers.
The Cost Breakdown
Before optimization, our RAG pipeline cost broke down as follows for 50,000 queries/month:
| Component | Monthly cost | What it covered |
|---|---|---|
| Embedding | $180 | text-embedding-3-large for all chunks |
| Pinecone | $70 | s1 pod |
| LLM inference | $940 | GPT-4 for every query |
| Total | ~$1,190 | Internal tool used by 40 engineers |
Optimization 1: Embedding Model Tiering
Not all content needs the same embedding quality. API documentation and code examples benefit from large models; general prose doesn't.
We switched to a tiered approach: text-embedding-3-small for general content (80% of our corpus), reserving text-embedding-3-large for code and structured data. Embedding cost dropped 65%.
Optimization 2: Query Caching
RAG queries from engineers cluster heavily. "How do I configure Airflow connections?" gets asked in 15 different phrasings. We added a semantic cache layer: hash the query embedding, check Redis for a cached response within cosine similarity 0.95, serve the cached answer.
Cache hit rate stabilized at 34% after two weeks of warmup. LLM calls dropped proportionally.
import numpy as np
import redis, json
from openai import OpenAI
client = OpenAI()
cache = redis.Redis(host="cache", port=6379)
SIM_THRESHOLD = 0.95
def embed_query(q: str) -> np.ndarray:
e = client.embeddings.create(model="text-embedding-3-small", input=q).data[0].embedding
return np.array(e, dtype=np.float32)
def cosine(a: np.ndarray, b: np.ndarray) -> float:
return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
def lookup(query: str):
q_emb = embed_query(query)
# Read recent N entries; in prod, use a vector index for the cache too.
for raw in cache.lrange("cache:recent", 0, 199):
entry = json.loads(raw)
if cosine(q_emb, np.array(entry["emb"], dtype=np.float32)) >= SIM_THRESHOLD:
return entry["answer"]
return None
def store(query: str, answer: str):
q_emb = embed_query(query).tolist()
cache.lpush("cache:recent", json.dumps({"emb": q_emb, "answer": answer}))
cache.ltrim("cache:recent", 0, 9999)Optimization 3: LLM Routing
The biggest win: not every query needs GPT-4. We added a complexity classifier (a fine-tuned text-classification model) that routes simple factual lookups to GPT-4o-mini and complex multi-step reasoning to GPT-4.
- GPT-4 — 20% of queries (complex reasoning, code generation)
- GPT-4o-mini — 80% of queries (factual lookups, summaries)
LLM cost dropped from $940 → $290/month.
Final Numbers
| Component | Before | After | Delta |
|---|---|---|---|
| Embedding | $180 | $63 | −65% |
| Pinecone | $70 | $70 | unchanged |
| LLM inference | $940 | $290 | −69% |
| Total | $1,190 | $423 | −64% |
Quality metrics (answer relevance, faithfulness) measured by RAGAS stayed within 3% of the pre-optimization baseline. We measured weekly to confirm the cost optimizations did not silently degrade the user experience.
Build production RAG
Production RAG is not the LangChain quick-start. It is tiered embedding, semantic caching, complexity-aware LLM routing, and the cost-attribution discipline to measure each component independently. The patterns generalize across domains and embedding providers.
Our RAG module covers the full stack: ingestion + chunking, embedding-model selection, hybrid scoring, semantic caching, LLM routing, RAGAS evaluation, and the cost dashboard that catches drift before it hits the budget.