How Our RAG Pipeline Works

I started with a naive embed-and-retrieve approach. Four steps: embed the query, search vectors, stuff into a prompt, generate. It worked in demos. It fell apart in production.

Users pasted invisible Unicode characters that broke pattern matching. Someone typed their BSN into a question about HR policy. A simple synonym mismatch meant the right document never surfaced. And then, on a Dutch government project, someone put [SYSTEM] tokens inside a SharePoint document and the LLM followed the instructions.

Each of those failures became a pipe. Four steps became eight, then twelve, then sixteen. The pipeline grew because production kept teaching me what I had missed.

This is the full architecture.

Why Laravel's Pipeline Pattern

I needed a way to add steps without rewriting the pipeline every time. Laravel's Pipeline pattern gave me exactly that - a chain of composable pipes, each receiving a shared Data object and passing it to the next.

Every pipe implements one interface: handle(Data $data, Closure $next). One method. One responsibility. The pipe does its work, mutates the Data object, calls $next($data).

I chose this over a monolithic service class for one reason: testability. I can unit-test SanitizeInput without a database connection. I can mock the LLM to test reranking logic. When I need a new step, I create a class and add one line to an array.

return Pipe::send($data)
    ->through([
        SanitizeInput::class,
        DetectPii::class,
        DetectPromptInjection::class,
        SetMcpServers::class,
        CheckRetrievalNeeded::class,
        QueryExpansion::class,
        QueryDecomposition::class,
        HybridSearch::class,
        DecomposedSearch::class,
        RerankChunks::class,
        FilterChunks::class,
        SanitizeChunks::class,
        SetSystemPrompt::class,
        SetUserPrompt::class,
        SetChatHistory::class,
        GeneratePendingRequest::class,
    ])
    ->then(fn (Data $data) => $data);

Sixteen classes. Readable top to bottom. Order matters.

The Data object starts with just a user input and an agent reference. By the time it exits, it carries an expanded query, decomposed sub-queries, retrieved chunks with scores, a system prompt, a user prompt, chat history, MCP server configurations, a Langfuse trace, and a fully assembled pending LLM request.

The key design decision: per-agent feature flags. Each pipe checks isFeatureEnabled() on the agent configuration before running. A customer support agent might need PII detection and strict prompt injection blocking. An internal engineering assistant might skip both and run with aggressive query expansion. Same pipeline, different behavior, zero code changes.

Multi-tenancy is baked in at the data layer. Every search query is scoped through a join chain: agent to knowledgebase to data source to document to chunk. There is no way to retrieve another tenant's data because the query never asks for it.

Phase 1: Input Security

Security runs first. Always. Before any LLM call, before any database query, before anything touches the user's input beyond reading it.

I learned this ordering the hard way. In an earlier version, query expansion ran before sanitization. Which meant malicious input reached the LLM before I had a chance to clean it.

SanitizeInput

The first pipe strips zero-width characters - U+200B (zero-width space), U+200C (zero-width non-joiner), U+200D (zero-width joiner), U+FEFF (BOM), and U+00AD (soft hyphen). These are invisible in most interfaces but can carry prompt injection payloads that bypass downstream pattern matching.

After stripping, the input goes through NFKC normalization via PHP's Normalizer class. NFKC decomposes characters and then recomposes them by compatibility, collapsing homoglyphs and variant forms into their canonical representations. The fullwidth "A" (U+FF21) becomes a regular "A". A ligature becomes its component characters.

Finally, the input is truncated to a configurable maximum length (default: 1000 characters).

Three operations. No LLM call. Microseconds of latency.

DetectPii

Users paste sensitive data into chat interfaces. It happens constantly. I added this pipe specifically for a Dutch government project where civil servants were asking questions about policy documents and typing citizen BSN numbers into the chat.

I use regex-based detection, not LLM-based detection. The reason: determinism. A regex either matches or it does not. No probability thresholds, no model drift, no false negatives that change between deployments. For compliance-critical data patterns, that predictability matters more than recall on edge cases.

LLM-based PII detection would make sense for unstructured PII like names or addresses. For structured national identifiers, the math is the validation.

Currently, I detect two Dutch PII types.

BSN (Burgerservicenummer) detection validates 9-digit sequences against the official 11-check algorithm - a weighted checksum that catches every valid BSN while ignoring random numbers. IBAN detection validates Dutch bank account numbers against the ISO 13616 mod-97 check. Both use regex to find candidates, then mathematical proof to confirm them. Full algorithm details.

Detected PII gets replaced with [VERWIJDERD] (Dutch for "removed") before the query moves forward. The Data object is flagged with piiDetected = true so downstream steps know the input was modified.

DetectPromptInjection

This is not a binary classifier. It is a scoring system.

The pipe runs the input against a set of configurable regex patterns covering English and Dutch injection phrases - things like "ignore previous instructions" and "negeer vorige instructies." Each pattern match adds 1 to the score.

Then the heuristics kick in. A base64-encoded payload (50+ alphanumeric characters matching the base64 pattern) adds 2. More than 10 special characters from the set {}[]<>|\ adds 1. Input exceeding 2000 characters adds 1.

The final score is compared against a per-agent configurable threshold. If the score meets or exceeds the threshold, the pipe throws a PromptInjectionDetectedException. Hard stop. No fallback. No degraded response.

I chose to hard-fail on prompt injection rather than degrade gracefully. If someone is trying to override the system prompt, I do not want a "polite decline." I want a wall. The alternative - letting a suspicious query through with reduced capabilities - is just a softer way of being compromised.

Why scoring over binary? Because different agents need different sensitivity. A public-facing customer support widget needs aggressive detection. An internal tool used by trusted employees can afford a higher threshold.

Phase 2: Query Understanding

The input is clean and safe. Now I need to make it searchable.

SetMcpServers

RAG answers questions from documents. But sometimes the answer is not in a document - it is in a database, an API, or a calculation.

Model Context Protocol lets the LLM call external tools during response generation. I built a Rechtspraak MCP server for Dutch case law so agents can pull court rulings in real time. The same pattern works for any external system - a pricing API, a project management tool, a CRM lookup.

The pipe itself is minimal. It loads MCP server configurations from the agent model and passes them to the Data object. The actual tool execution happens later, during generation, via Prism Relay. Prism translates tool definitions into the format each LLM provider expects and routes tool calls back to the configured servers.

MCP servers are configured per agent. A legal agent gets the Rechtspraak server. A sales agent gets the CRM connector. No code changes - just configuration.

No LLM call in this pipe. Just loading configuration for later.

CheckRetrievalNeeded

When someone types "hi" or "good morning," searching across thousands of document chunks is pointless. This pipe short-circuits retrieval for simple greetings.

The check is minimal: word count at most 2, and the input appears in a configurable greeting list. When matched, the pipe clears chunks and documents on the Data object, which causes all downstream search steps to skip. The agent responds conversationally instead of retrieving documents.

This is not an optimization. It is user experience. Nobody wants to wait 3 seconds for a semantic search just to get a "hello" back.

QueryExpansion

Here is where the first LLM call happens. Of all the steps I added to the pipeline, this one surprised me the most. Not reranking. Not better embeddings. Not a fancier vector database. Just asking the LLM to add synonyms and related terms. I do not have formal benchmarks yet - when I do, I will publish them here. But in my testing, query expansion consistently surfaced documents that the original query missed entirely.

Query expansion enriches the original input with synonyms, abbreviations, related terms, and alternative phrasings. A user searching for "remote work policy" gets an expanded query that also covers "flexible working," "work from home," "WFH," "telecommute" - terms that might appear in the actual documents.

The expansion is LLM-driven, not dictionary-based. An LLM understands that "PTO" and "paid time off" and "leave policy" are related in a way no static synonym map can capture.

Per-agent customization matters here. An agent serving a legal department can provide custom expansion instructions and worked examples. "When the query mentions 'contract termination,' also include 'opzegging,' 'beeindiging,' and 'ontbinding.'" This is especially powerful for multilingual knowledge bases where documents are in Dutch but users search in English, or the other way around.

The expanded query is stored separately on the Data object. The original input is preserved. getSearchQuery() returns the expanded version when available, falling back to the original.

If the LLM call fails, the original query is used unchanged. The pipeline does not break, it just searches with less enrichment.

QueryDecomposition

Complex questions need to be broken apart before they can be answered well.

"What is the pricing for enterprise plans and how do I contact sales?" is really two questions. Searching for both in a single query dilutes the results. Query decomposition uses an LLM call to split multi-part questions into independent sub-queries.

The LLM returns a JSON array of sub-queries, or an empty array if the question is simple enough to search as-is. Decomposition is capped at a configurable maximum (default: 5 sub-queries) to prevent runaway costs.

Decomposition operates on the expanded query when available, not the raw input. This ordering is deliberate. Expanding the query first means the decomposition LLM works with richer input, producing better sub-queries. Decomposing first would mean expanding each sub-query separately - more LLM calls, less context per call.

Phase 3: Retrieval

Two search strategies. Two post-processing layers. One sanitization pass.

HybridSearch

The first time I ran pure vector search on Dutch government documents, BM25 from 1994 outperformed it. That was humbling.

Pure semantic search is brittle. "Verlofregelingen" (Dutch for leave regulations) does not embed close to "leave policy" in most English-trained models. Pure lexical search has poor recall on paraphrased questions. I run both and merge.

Lexical search runs ILIKE queries against chunk content in PostgreSQL. Before querying, the input goes through stopword removal (via the StopWords package) and LIKE-wildcard escaping to prevent SQL injection through % and _ characters. Each query term becomes a separate ILIKE clause ORed together.

Scoring is term-frequency based: count how many times each query term appears in the chunk content, divide by the number of terms, multiply by the agent's lexical_search_weight.

Semantic search uses pgvector's cosine distance operator (<=>). The query is embedded on-the-fly via the configured embedding service, and the resulting vector is compared against stored chunk embeddings. I support both 768 and 1024 dimension embeddings - the pipe selects the correct column based on the embedding dimension.

I use pgvector in PostgreSQL rather than a separate vector database. One less service to deploy, one less thing to break. PostgreSQL handles my relational data, my full-text search, and my vector search. For the scale I operate at, this simplicity beats the marginal performance gains of a dedicated vector store.

Semantic scoring follows the formula: 1 / (1 + distance) multiplied by the agent's semantic_search_weight. This transforms the raw cosine distance (where 0 is identical and 2 is opposite) into a 0-to-1 relevance score.

Both searches retrieve top_k * 2 candidates to ensure sufficient coverage before merging. The merge deduplicates by chunk ID, sorts by score descending, and takes the final top_k_retrieved_chunks.

The search weights are per-agent configurable. You can run pure lexical (semantic weight = 0), pure semantic (lexical weight = 0), or any blend. Some knowledge bases work better with heavier lexical weighting - especially when the content uses domain-specific terminology that embedding models have not seen enough of during training.

Semantic search gracefully degrades on failure. If the embedding service is down or the query fails, the pipe logs the error and returns an empty collection. Lexical results still flow through.

DecomposedSearch

When query decomposition produced sub-queries and the initial HybridSearch found zero results, DecomposedSearch kicks in. It runs HybridSearch once per sub-query.

Results are merged across sub-queries: when the same chunk appears in results for multiple sub-queries, it keeps the highest score. This rewards documents that are relevant to multiple parts of a complex question.

DecomposedSearch only runs when two conditions are met: decomposed queries exist AND the primary HybridSearch found nothing. This prevents double-searching when the original query already yielded good results. Each sub-query triggers a full HybridSearch - lexical queries, embedding generation, pgvector distance calculation. That cost adds up, which is why it is a fallback, not the default path.

RerankChunks

Retrieved chunks are sorted by retrieval score, but retrieval scores are noisy. A chunk might rank high on lexical match because it contains the right keywords without actually answering the question.

LLM-based reranking scores each chunk on a 0.0 to 1.0 scale for relevance to the original user query. The scoring rubric is built into the prompt:

0.9-1.0: directly answers the query
0.7-0.89: very relevant, contains important information
0.5-0.69: moderately relevant, provides useful context
0.3-0.49: somewhat relevant, tangentially related
0.0-0.29: not relevant

The LLM returns a JSON object mapping chunk indices to scores. Chunks below the agent's reranker_score_filter_threshold are dropped. The remaining chunks are sorted by rerank score and cut to top_k_reranked_chunks.

Why LLM reranking instead of a dedicated cross-encoder model? I chose it because I can customize the scoring prompt per agent and I do not need to deploy and maintain a separate model. A compliance agent can provide a rerank prompt that prioritizes regulatory documents over informal guidance. An engineering agent can weight API documentation over meeting notes. A cross-encoder gives you one scoring function. An LLM reranker gives you a different one per agent, defined in plain language.

The trade-off is latency - an LLM call is slower than a cross-encoder inference. For my use cases, the flexibility wins.

If the LLM call fails, the pipe falls back to the original retrieval scores. Better to rank by retrieval score than to drop all chunks.

FilterChunks

Reranking orders by relevance. Filtering removes the irrelevant tail.

This is a stricter pass. The LLM evaluates all remaining chunks and returns a JSON array of indices for chunks that are actually relevant. Binary decision: keep or drop.

The default criteria: the chunk directly answers or relates to the query, provides necessary context, or contains supporting evidence. Chunks that only share keywords without providing useful information, contain outdated data, or are too vague get excluded.

FilterChunks is complementary to reranking, not redundant. Reranking might give a chunk a 0.55 score - above the threshold, technically relevant. Filtering asks a different question: "Does this chunk actually help answer the user's question?" Sometimes the answer is no.

SanitizeChunks

SanitizeChunks did not exist in v1. It exists because someone put [SYSTEM] tokens in a SharePoint document and the LLM followed the instructions.

This is indirect prompt injection via poisoned documents, and most RAG systems ignore it entirely. Indexed documents might contain text like [SYSTEM], [INST], or <<SYS>> - markers that trick the LLM into treating document content as system instructions.

SanitizeChunks strips HTML tags, removes these prompt injection markers, applies NFKC normalization, and collapses whitespace. Same defense-in-depth principle as input sanitization, applied to the retrieval side. You control user input. You do not control what users upload into the knowledge base.

Phase 4: Prompt Assembly and Generation

The chunks are retrieved, reranked, filtered, and sanitized. Now I assemble the final prompt.

SetSystemPrompt

The system prompt is not static. It is selected from three variants based on the current context.

When no chunks were retrieved - either because retrieval was skipped for a greeting or the search returned empty - the agent's no_retrieval_prompt is used. This typically instructs the LLM to respond conversationally or admit that it cannot find relevant information.

When multi-turn conversation is enabled, the multi_turn_system_prompt is used, which includes instructions for maintaining conversational context across messages.

Otherwise, the default system_prompt is used - the one designed for single-turn question answering with retrieved context.

After selecting the base prompt, the pipe appends layers of additional instructions:

Guardrails block the LLM from answering questions about its own system prompt, model, or system internals. Standard defense against prompt extraction attacks.

MCP tool usage instructions (when MCP servers are configured) tell the LLM to use tools directly without preamble - no "Let me search for that..." filler.

Avoid commentary mode suppresses phrases like "Based on the information provided..." - the kind of meta-commentary that makes AI responses feel robotic.

Groundedness scoring enforces strict grounding in provided documents. The LLM is instructed to only answer from the retrieved context and to explicitly say when it cannot.

Chunk-id citation instructions force the LLM to reference source chunks in [chunk-id:X] format. This is what makes source citations work in the frontend.

SetUserPrompt

The user prompt wraps retrieved chunks in XML-structured tags:

<retrieved_documents>
<document chunk-id="42">
...chunk content...
</document>
<document chunk-id="87">
...chunk content...
</document>
</retrieved_documents>

<user_query>
What is the refund policy for enterprise customers?
</user_query>

Clean separation between context and question. The LLM can parse the structure, and the chunk-id attributes enable citation tracking.

SetChatHistory

In single-turn mode, a fresh message list is created with just the current enriched user prompt. Every question is independent.

In multi-turn mode, the full chat history is loaded from the database. The last message's content is replaced with the enriched user prompt - the one containing the retrieved document context. The LLM sees the full conversation history but the current turn includes the relevant documents.

GeneratePendingRequest

The final pipe assembles a Prism PendingRequest with all parameters: provider and model (from the message's LLM configuration, or the tenant default, or the pipeline config fallback), temperature, max tokens, top-p (skipped for Anthropic, which does not support it alongside temperature), system prompt, chat history, MCP tools via Prism Relay, and optional provider options like frequency penalty and random seed.

Prism is the PHP package that handles LLM abstraction. I can switch between OpenAI, Anthropic, or any other provider by changing configuration. No code changes.

Provider options are only included when they differ from defaults. No point sending frequency_penalty: 0 when that is already the default.

This pipe does not execute the request. It returns a pending request object. The streaming controller handles execution separately. This is an intentional architectural boundary: the pipeline handles retrieval and assembly, the controller handles generation and streaming. The pipeline can be tested without streaming. The streaming layer can be changed without touching retrieval logic.

Observability with Langfuse

I maintain the axyr/laravel-langfuse package, so I built the observability integration exactly the way I wanted it. Every pipe reports to a shared Langfuse trace, initialized at the pipeline entry point.

Three instrumentation levels map to three types of pipeline work.

Events are lightweight logs for non-LLM steps. SanitizeInput logs whether the input was modified and its final length. SetSystemPrompt logs which prompt variant was selected. SetChatHistory logs the message count. No timing, no token counts - just structured metadata.

Spans track timed operations. HybridSearch records the query, weights, and top-k configuration at the start, then closes the span with result counts and the top score. GeneratePendingRequest records the final provider, model, temperature, and token limit. Spans answer: "How long did this step take?"

Generations provide full LLM call tracking with model, input, output, and token usage. Query expansion, query decomposition, reranking, and filtering are all traced as generations. This is where you see exactly what prompt was sent, what came back, and what it cost.

When Langfuse is not configured, the Data object initializes with a NullLangfuseTrace - a null-object implementation that accepts all the same method calls but does nothing. Zero overhead. No conditionals scattered through the pipe code. The pipes call logEvent, startSpan, and startGeneration regardless.

This observability layer enables latency analysis per pipe, token cost attribution per step, retrieval quality debugging (which chunks surfaced and what scores they got), and prompt iteration tracking over time.

What I Learned Building This

The pipeline is not something I designed on a whiteboard. It grew from production failures. Each step has a story.

SanitizeInput exists because invisible Unicode characters bypassed my prompt injection patterns. DetectPii exists because a government client needed BSN protection before I could deploy. SanitizeChunks exists because indirect prompt injection through documents is a real attack vector that most RAG tutorials never mention. DecomposedSearch exists because complex multi-part questions returned nothing when searched as a single query.

The order is deliberate too. Security before LLM calls, expansion before decomposition, retrieval before reranking. I tried other orderings. They produced worse results or security gaps.

Four LLM calls before generation even starts is expensive. Query expansion, decomposition, reranking, filtering - each adds a round-trip. But feature flags are the performance lever. Disable expansion and decomposition for a latency-sensitive agent and you cut two LLM round-trips. Disable reranking and filtering and you cut two more. For simple knowledge bases with clean, well-structured content, the quality loss is minimal.

The architecture lets each deployment find its own balance. A public FAQ with 50 documents does not need the same pipeline as an enterprise knowledge base with 500,000 chunks across SharePoint, Jira, and Teams.

The commented code at the bottom of the pipeline hints at what comes next: user fact extraction and query rewriting. Same pattern. New pipes. No changes to existing code.

That is what the architecture is for.

Learn about our security model | See observability features | Explore data source integrations