Retrieval Augmented Generation Tutorial: Production RAG

Building a production retrieval augmented generation pipeline requires pairing hybrid vector and document indexing with re-ranking filters and matched generation models. Synthesis costs vary dramatically across frontier tiers: OpenAI GPT-6 Luna costs $0.60 per 1 million input plus 1 million output tokens, while GPT-6 Astra costs $60, making multi-stage retrieval architecture essential to keep token consumption bounded.12

Core stages and evaluation metrics of a RAG pipeline

A standard RAG pipeline operates across two foundational stages: data indexing, where raw data is ingested and split before an embedding model converts chunks into vectors, followed by retrieval and generation. Preparing clean ingestion data often involves generating structured markdown documentation via tools like the Firecrawl llms.txt generator before chunking routines tune overlap to prevent context loss across boundaries.3

Evaluating retrieval augmented generation systems requires measuring retrieval accuracy separate from generation quality. Pipelines track five standard performance metrics to balance relevance and fidelity: Recall@k, Mean Reciprocal Rank (MRR), context precision, faithfulness, and hallucination rate.3

Hybrid indexing limits in Pinecone and Weaviate

In Pinecone, new indexes default to document indexes while vector indexes remain fully supported. A single Pinecone document index schema can combine dense_vector, sparse_vector, and full_text_search-enabled string fields, where full-text search executes BM25 token matching using Lucene query syntax without requiring an external machine learning model.4

Operating sparse vectors in Pinecone introduces strict system boundaries. Sparse vectors permit a maximum of 2,048 non-zero values per vector and enforce limits of 10 maximum upserts per second and 100 maximum queries per second per index. During matching operations, Pinecone automatically discards any sparse vectors where the dotproduct score is 0.4

Pinecone caps query execution at a maximum top_k value of 10,000. At the tenant layer, serverless indexes scale from 100 namespaces on the Starter tier up to 1,000,000 namespaces on the Enterprise tier. Metadata fields must be configured as flat key-value pairs in JSON; nested JSON objects are unsupported.4

For self-hosted or alternative deployments, Weaviate supports dynamic indexing, which maintains a flat vector index while the dataset is small and automatically switches to HNSW as the corpus expands, alongside standalone flat and HNSW modes. Inverted indexes in Weaviate are deployed alongside vector configurations to run BM25 queries or accelerate metadata filtering operations.5

Candidate scoring with Cohere Rerank

Multi-stage retrieval routes top vector and BM25 candidates through a re-ranking model to filter out irrelevant passages before passing context to an LLM. When using Cohere rerank-v4.0-pro, rerank-v4.0-fast, rerank-v3.5, or rerank-v3.0, the endpoint raises an error if an application passes more than 10,000 documents in a single request. Cohere normalizes all output relevance scores to a float range between 0 and 1, where values approaching 1 reflect high relevance.6

The Cohere rerank-v4.0 family expands total context length to 32,768 tokens, breaking documents into 32,764-token chunks after reserving four tokens for special markers. The query context budget permits queries up to 16,384 tokens, truncating anything larger to leave the remaining 16,384 tokens for document text. In contrast, legacy rerank-v3.0 and rerank-v3.5 models operate on a 4,096-token context window, chunking documents at 4,093 tokens and truncating queries at 2,048 tokens. All of these Cohere models accept structured inputs formatted as a list of YAML strings.6

Orchestrating agentic RAG workflows

When retrieval tasks require iterative exploration rather than single-step queries, agentic architectures break down operations into specialized roles: Planner, Retriever, Synthesizer, and Verifier. Implementing structured AI agent orchestration allows systems to inspect intermediate candidate chunks and trigger additional searches before finalizing responses.3

In LangChain Deep Agents, workflows orchestrate retrieval patterns such as skills-guided retrieval alongside todo-driven investigation. For automated validation, Deep Agents supports beta rubric-checked grounding, which requires deepagents>=0.6.5 to evaluate source grounding during synthesis.7

Cost trade-offs across OpenAI GPT-6 models

Inference synthesis accounts for the largest operational expense in a production RAG pipeline. Standard processing for OpenAI GPT-6 Astra under a 272K context window costs $10.00 per 1M input tokens, $1.00 per 1M cached input tokens, and $50.00 per 1M output tokens. For intermediate reasoning workloads, GPT-6.1 Sol costs $2.00 per 1M input tokens, $0.10 per 1M cached input tokens, and $10.00 per 1M output tokens.1

For cost-sensitive document extraction, OpenAI GPT-6 Luna reduces expenses to $0.10 per 1M input tokens, $0.01 per 1M cached input tokens, and $0.50 per 1M output tokens. Data collected on 2026-10-13 confirms that across 1 million input plus 1 million output tokens, GPT-6 Luna costs $0.60 compared to $60 for GPT-6 Astra, representing an exact 100x cost differential. Pipelines running non-urgent background batch synthesis can leverage OpenAI's Batch API to receive a 50% discount on inputs and outputs for asynchronous jobs that complete within 24 hours.12

If a pipeline incorporates external live retrieval tools alongside vector storage, OpenAI charges $10.00 per 1k calls for its Web search tool, while search content tokens returned by the tool are provided for free.1