Knowably.ai
Early access
Features Integrations Security Pricing Custom Solutions

Language

English Nederlands
Articles Technical

Observability

Structured tracing for every query through the RAG pipeline. See which documents were retrieved, what the LLM received, and where latency occurs.

The first time a customer told me an answer was wrong, I had nothing. No way to see which documents were retrieved, which pipe took the longest, what the LLM actually received as context. Just a user message and a bad answer, with ten pipeline steps in between.

That is when I knew Knowably needed tracing. Not metrics dashboards. Not log files. Structured traces that show every step of every query, from raw input to streamed response.

I built and maintain axyr/laravel-langfuse because I needed this for Knowably. The existing PHP Langfuse packages did not match how I structure Laravel applications. So I wrote one that does. It speaks Laravel natively - facades, DTOs, typed objects - and it fits into the pipeline architecture without friction.

Why Langfuse

I evaluated several observability tools before landing on Langfuse. What sold me: it was purpose-built for AI applications. Traces, spans, generations, events, scores - it speaks the language of LLM pipelines. Not generic APM retrofitted for AI. The data model matches how a RAG pipeline actually works.

Every chat request in Knowably - whether from an authenticated user or an embed widget visitor - creates a Langfuse trace. The trace captures the full lifecycle from raw user input through retrieval to streamed response.

One design decision I am proud of: the integration is non-intrusive. If Langfuse is unavailable or not configured, the pipeline runs without it. The Data object that flows through the pipeline initializes with a NullLangfuseTrace - a null-object implementation that accepts all trace calls and silently discards them.

public function __construct()
{
    $this->langfuseTrace = new NullLangfuseTrace;
}

No conditionals scattered through pipe code. No risk of a monitoring failure breaking your chat. When Langfuse is available, the real trace replaces this default before the pipeline runs. Every pipe instruments itself the same way regardless.

I chose this approach after dealing with monitoring code that brought down production in a previous project. Observability should never be a failure point.

Trace Initialization

A trace begins at the start of each chat stream request, before the pipeline runs. For authenticated chats, the trace name is chat. For embed widget chats, it is embed-chat. This separation makes filtering trivial in the Langfuse dashboard - I can immediately isolate widget traffic from internal users.

Trace metadata includes user ID, chat session ID, agent name, tenant, LLM model, and provider. But the tags are where it gets interesting. Tags are built dynamically from the agent's feature flags:

$features = [
    'rerank_enabled' => 'rerank',
    'filter_enabled' => 'filter',
    'query_expansion_enabled' => 'query-expansion',
    'query_decomposition_enabled' => 'query-decomposition',
    'prompt_injection_detection_enabled' => 'injection-detection',
    'pii_detection_enabled' => 'pii-detection',
    'multi_turn_enabled' => 'multi-turn',
];

I added this because I kept wanting to answer questions like "are reranked queries slower?" or "does query decomposition improve feedback scores?" Without tags, answering that means writing database queries. With tags, it is one filter click in Langfuse.

Per-Pipe Instrumentation

Every pipe in the RAG pipeline inherits from AbstractPipe, which provides three instrumentation methods. Choosing the right one for each pipe matters.

Spans track operations that have duration but do not call an LLM. Search operations, detection logic - things where wall-clock time is the metric that matters.

Generations track operations that call an LLM and consume tokens. Query expansion, reranking, chunk filtering. These record input, output, model, and token usage. This is where cost lives.

Events log lightweight state changes. No duration, just a point-in-time snapshot. Sanitization results, prompt assembly details - things I want to confirm happened without measuring how long they took.

The base class keeps it simple:

protected function startSpan(Data $data, string $name, mixed $input = null): LangfuseSpan
{
    return $data->langfuseTrace()->span(new SpanBody(
        name: $name,
        input: $input,
    ));
}

Because langfuseTrace() always returns either a real trace or the null object, pipes never check whether tracing is active. Three lines to instrument any operation.

Pipe-by-Pipe Tracing Breakdown

Here is exactly what each pipe logs and why I chose to log it.

sanitize-input (event)

Logs whether the input was modified by sanitization and the final character length. If zero-width characters were removed or Unicode was normalized, modified: true tells you immediately. I log this because sanitization bugs are silent - you only notice when downstream search returns nothing because invisible characters corrupted the query.

detect-pii (event)

Logs whether PII was detected and, if so, which types were found. When a Dutch BSN or IBAN is stripped from the input, the event records pii_detected: true with the specific pii_types array. When nothing is found, it still logs pii_detected: false. I always log the negative case too - so I can confirm the check actually ran, not that it was skipped.

detect-prompt-injection (span)

This is a span, not an event, because the scoring logic has measurable duration. Logs the computed injection score, the agent's configured threshold, and whether the input passed.

$span->end(output: [
    'score' => $score,
    'threshold' => $threshold,
    'passed' => $passed,
]);

I log the score even when it passes. When you tune thresholds later, you need the historical data. Without it, you are changing a number based on gut feeling instead of looking at the distribution of scores across real queries.

query-expansion (generation)

Logs the original user input as generation input, the expanded query as output, the model used, and full token usage. If expansion fails, the generation records the error in statusMessage and the pipeline continues with the original query. I have seen expansion produce worse queries than the original - the trace makes that visible.

query-decomposition (generation)

Logs the search query (which may already be the expanded query from the previous pipe), the decomposed sub-queries as output, the model, and token usage. An empty output array means the LLM determined the query was simple enough not to decompose. I watch for queries that get decomposed into five or six sub-queries - that is usually the LLM overthinking a straightforward question, and it multiplies cost.

hybrid-search (span)

The richest span in the pipeline. Input captures the query, lexical weight, semantic weight, and top_k setting. Output records total chunks found, lexical result count, semantic result count, and the top score.

This is where I diagnose retrieval quality. If semantic_results: 0, the embeddings might be stale. If lexical_results: 47 but total_chunks: 10, deduplication and scoring are working as intended.

rerank-chunks (generation)

Input records the chunk count entering reranking and the user query. Output logs the filtered chunk count, top rerank score, and the score threshold. Token usage captures the cost of the LLM-based relevance scoring.

When reranking drops 15 chunks to 3, this generation shows you exactly what happened. I have used this to catch a misconfigured threshold that was throwing away relevant results.

filter-chunks (generation)

Similar structure to reranking. Logs input_chunk_count and output_chunk_count. If filtering removes everything, you see output_chunk_count: 0 - a clear signal. I have had this happen when the filter prompt was too strict for a particular knowledge base. Without the trace, it just looked like "no answer."

set-system-prompt (event)

Logs the prompt type (default, multi_turn, or no_retrieval) and the prompt length in characters. The prompt type tells you which system prompt template was selected. If you see no_retrieval when you expected documents, the issue is upstream in search - do not waste time looking at the prompt.

set-user-prompt (event)

Logs the chunk count, chunk IDs, document IDs, and total prompt length. The chunk IDs and document IDs create a direct link between the trace and the specific content the LLM received. I added document IDs after spending too long manually matching chunks back to their source documents during a debugging session.

set-chat-history (event)

Logs whether multi-turn is enabled and how many messages are in the conversation history. A high message count in multi-turn mode correlates with longer prompts and higher token costs. I use this to spot conversations where context windows are getting bloated.

Chat Completion Generation

After the pipeline finishes, the final LLM call is tracked as a separate generation named chat-completion on the trace. This is the most expensive step - the actual answer generation.

It records the model, temperature, and max_tokens parameters. Input metadata captures system prompt length, user prompt length, and chunk count. These three numbers tell you how large the context window is for this request.

Token usage comes from the Prism SDK's StreamEndEvent. Because Knowably streams responses, token counts are only available after the stream completes:

$generation->end(
    output: $cleanedParts['text'] ?? '',
    usage: $streamUsage ? new Usage(
        input: $streamUsage->promptTokens,
        output: $streamUsage->completionTokens,
    ) : null,
);

The trace itself is also updated with the final output text. So every trace is a complete record from input to answer.

End-to-End Trace Linking

Every assistant message stores the Langfuse trace ID directly in the database via the langfuse_trace_id column on the chat_messages table.

This was one of the first things I implemented after getting basic tracing working. Without it, you are searching Langfuse by timestamp and hoping you find the right trace. With it, any message a user sees links directly to the full pipeline execution that produced it.

Support gets a user complaint about a bad answer? Look up the message, follow the trace ID, open it in Langfuse. Every pipe that ran, every LLM call that was made, every chunk that was retrieved. This works for both authenticated and embed widget conversations.

User Feedback Loop

When a user upvotes or downvotes a response, Knowably sends that feedback to Langfuse as a numeric score attached to the trace.

Langfuse::score(new ScoreBody(
    name: 'user_feedback',
    traceId: $message->langfuse_trace_id,
    value: $message->is_upvoted ? 1.0 : 0.0,
    dataType: ScoreDataType::NUMERIC,
));

Score name is user_feedback. Value is 1.0 for upvote, 0.0 for downvote.

A second score - retrieval_chunk_count - is recorded automatically on every trace. This captures how many chunks were used in the response, enabling retrieval quality analysis without opening individual traces.

The combination is powerful. In Langfuse, I filter traces by low feedback scores and immediately inspect the retrieval steps, reranking decisions, and generation parameters that produced the poor answer. This closes the loop between user experience and pipeline debugging. No guessing involved.

Token Usage Tracking

Per-Generation Counts

Every LLM call in the pipeline - query expansion, query decomposition, reranking, chunk filtering, chat completion - records input and output token counts. These are captured from the Prism SDK response and passed to Langfuse as Usage objects.

For non-streaming calls (expansion, decomposition, reranking, filtering), usage comes from the TextResponse. For the streamed chat completion, usage comes from the StreamEndEvent that fires when the stream finishes.

Provider-Level Aggregation

Knowably's dashboard tracks token usage broken down by provider. Daily token consumption trends show whether costs are climbing. Monthly totals roll up at the workspace level.

Cost Visibility

Monthly cost is calculated and displayed on the dashboard. When a CTO asks "how much is this costing us?", the answer is one dashboard load away. I built this because every AI project I have seen eventually gets the cost question, and "let me check" is not a good answer.

Dashboard Metrics

The built-in dashboard surfaces the data that matters for daily operations:

Overview cards show agent count, data source count, total queries, total tokens, storage usage, and monthly cost. One glance tells you whether the system is healthy.

Trend charts track daily query volume, daily token consumption, storage by category, and token usage by provider. These reveal patterns. A sudden spike in token usage might mean a new agent is configured with aggressive query decomposition - something I have caught this way more than once.

Performance metrics cover average response time, success rate, top agents by usage, and error type distribution.

The dashboard auto-refreshes every 30 seconds. All metrics are scoped to the current workspace - tenant isolation applies to analytics too.

Self-Hosted Langfuse Deployment

Some organizations cannot send observability data externally. Government clients, healthcare, finance - sending user queries and LLM responses to a third-party SaaS is often a non-starter. The same Langfuse that runs on langfuse.com runs on your Docker host.

Knowably ships a complete Langfuse stack in its docker-compose.yml. No external SaaS dependency required.

The stack includes six services: langfuse-web (the UI and API, v3), langfuse-worker (background job processing, v3), PostgreSQL (metadata storage), ClickHouse (high-performance analytics), Redis (job queues), and MinIO (S3-compatible object storage for events and media).

ClickHouse runs in single-node mode (CLICKHOUSE_CLUSTER_ENABLED=false) for development and small deployments. For larger installations, cluster mode is a configuration flag away.

Automatic Provisioning

I wanted zero manual Langfuse UI configuration on first run. Environment variables seed the initial organization, admin user, project, and API keys:

LANGFUSE_INIT_ORG_ID=knowably
LANGFUSE_INIT_PROJECT_ID=knowably
[email protected]
LANGFUSE_INIT_PROJECT_PUBLIC_KEY=...
LANGFUSE_INIT_PROJECT_SECRET_KEY=...

Run docker-compose up and the application connects immediately. Reproducible, automated, no clicking through setup wizards. I have set up Langfuse this way for development, staging, and production environments without touching the UI once.

Data Sovereignty

Self-hosted Langfuse means all trace data stays within your infrastructure. No LLM inputs, outputs, or user queries leave your network for observability purposes.

For organizations in regulated industries, this is not a nice-to-have. It is a requirement. Knowably's self-hosted approach means GDPR compliance is the default, not an upgrade tier.

Tenant isolation extends to trace metadata. Each trace includes the tenant ID, so multi-tenant deployments can filter traces by workspace without risk of data mixing.

Debugging Workflows

These are real scenarios I have encountered. Not hypotheticals.

Bad answer reported. A customer flagged a response as wrong. I looked up the message in the database, followed the langfuse_trace_id to the full trace. The hybrid-search span showed 12 chunks retrieved, but the rerank-chunks generation had dropped all of them below threshold. The reranker was too aggressive for that particular knowledge base. I adjusted the threshold, and the same query returned a correct answer. Without the trace, I would have been reading logs and guessing.

Latency spike. Dashboard showed response times climbing on a Monday morning. I opened recent traces and compared span durations. hybrid-search was taking 3 seconds instead of the usual 200ms. Turned out the vector database was running a vacuum operation. The traces made the diagnosis instant - I did not have to reproduce the problem.

Cost spike. Monthly token usage jumped 40% after a new customer onboarded. I filtered Langfuse by the query-decomposition tag and found their agent was decomposing every query into five sub-queries. Each sub-query triggered its own search and reranking cycle. The per-generation token counts made the source obvious. We tuned the decomposition prompt to be more conservative.

Quality degradation. Filtered traces by user_feedback scores of 0.0 and noticed a pattern: most downvoted answers came from one agent. The set-user-prompt events showed chunk_count: 0 on many traces. The data source had gone stale - new documents were not being indexed. The feature flag tags let me confirm it was isolated to that agent, not a platform-wide issue.

Related Pages