Agent Configuration Guide
A comprehensive guide to configuring Knowably AI agents, covering all settings from system prompts and retrieval parameters to guardrails and output formatting.
Start with the defaults. Every setting exists because the default was not right for someone.
Knowably agents expose thirty-seven fields across ten sections. Each one maps to a specific stage of the RAG pipeline. Most of them you will never touch. The ones you do touch will determine whether your agent gives precise, trustworthy answers or hallucinates confidently.
This is the field-by-field reference. If you have not read the AI Agents feature overview, start there first.
Two creation flows exist. Quick create runs auto-configuration. Advanced create gives full control from the start. Both land you in the same editor.
Agent Editor Overview
The sidebar groups sections by function.
General - identity, knowledge scope, external tools. Base - System Prompts and Query Understanding. Advanced - Retrieval, Rerank, Filter, Generation, User Experience, Query Reformulation. Security - Guardrails. Integrations - Embed widget.
Each section maps to a pipeline stage. The editor is your control surface for the RAG pipeline.
An agent has two statuses: "configuring" (auto-configuration running) and "ready" (operational). You can only chat with agents in the "ready" state.
1. General
Identity and scope. Get this right before touching anything else.
name
Type: string, required
Display name shown in the agent list, chat headers, and embed widget. "HR Policy Assistant" beats "Agent 1."
description
Type: string, nullable
Short summary shown in the agents list. Auto-generated during quick create (see section 12). Two to three sentences is plenty.
knowledgebases
Type: array of knowledge base IDs, nullable
Which knowledge bases this agent searches. An agent can connect to multiple knowledge bases - retrieval runs across all of them simultaneously.
mcp_servers
Type: array of server names, nullable
Connects external tools via Model Context Protocol. Only appears when MCP servers are configured in your workspace. Not supported with AWS Bedrock models - requires Anthropic API models.
max_tool_steps
Type: integer, range 1-10, default 10
Only visible when MCP servers are selected. Controls the maximum tool execution iterations in agentic loops. Higher values allow more complex multi-step reasoning but increase latency and cost.
2. System Prompts
The most underrated section. A mediocre system prompt with perfect retrieval settings still produces mediocre answers.
system_prompt
Type: string, required
The primary instruction set sent to the LLM during generation, active when the pipeline has retrieved relevant documents. Controls tone, formatting, terminology, and behavioral boundaries.
The default instructs the agent to answer strictly from retrieved sources and refuse gracefully when it lacks information. Customize this for your domain - industry terminology, formatting preferences, persona. This is where you make the agent yours.
no_retrieval_prompt
Type: string, required
Used when no retrieval is needed (greetings, off-topic queries) or when no relevant documents were found. This is your fallback personality. A well-written no-retrieval prompt prevents the agent from going silent when it has no documents to reference.
3. Query Understanding
Pre-processing that interprets the user's query before retrieval starts. These settings have an outsized impact on conversation quality.
multi_turn_enabled
Type: boolean, default true
Enables conversational context awareness. Follow-up queries like "What about the deadline?" get reformulated into self-contained queries like "What is the deadline for the Q3 compliance report?" using conversation history. Without this, each query is treated in isolation.
I leave this on for every agent. Real users ask follow-up questions.
multi_turn_system_prompt
Type: string, required (visible only when multi_turn_enabled is true)
System prompt for the reformulation LLM call. The default handles pronoun resolution, missing subjects, temporal references, and new-topic detection. Includes a {{ date }} template variable for relative date handling.
Customize when your domain has specific entity resolution needs - project names, product codes, or abbreviations that users reference across turns.
retrieval_check_enabled
Type: boolean, default true
Uses an LLM call to classify whether the query needs document retrieval. "Hi there" skips retrieval and uses the no_retrieval_prompt. When disabled, every query runs through the full pipeline.
Disable if your agent should always attempt retrieval. Keep it on to save latency and cost on greetings and off-topic queries.
4. Query Reformulation
Expands or decomposes queries to improve retrieval recall. Powerful, but each step is a separate LLM call. Use deliberately.
query_expansion_enabled
Type: boolean, default false
Generates alternative phrasings and related terms. "Remote work rules" expands to include "telecommuting policy," "work from home guidelines," and "flexible working arrangement." Increases recall by searching with multiple reformulations simultaneously.
query_expansion_instructions
Type: string, nullable
Free-text instructions for the LLM that performs expansion. Use this for domain-specific guidance: "Include Dutch legal terminology equivalents" or "Expand product names to include both internal codenames and marketing names."
query_expansion_examples
Type: array, nullable (entered as multi-line text, one per line)
Few-shot examples guiding expansion behavior. Show the input query and expected expansions. The more representative of actual user queries, the better.
query_decomposition_enabled
Type: boolean, default false
Breaks complex multi-part queries into simpler sub-queries. "What is our vacation policy and how do I request time off?" becomes two retrieval queries. Results are merged before reranking. Enable when users tend to ask compound questions.
query_decomposition_examples
Type: array, nullable (entered as multi-line text, one per line)
Few-shot examples for decomposition. Show the complex input and expected sub-queries.
How expansion and decomposition interact
Both can be enabled simultaneously. Expansion runs first, then decomposition operates on the expanded query. This means the decomposition model works with richer input - synonyms and related terms are already present when it decides how to split.
For most use cases, pick one or the other. I reach for expansion more often. Decomposition shines when users consistently ask compound questions, which is less common than you'd expect.
5. Retrieval
Controls how documents are fetched from the vector store and lexical index. This is where answer quality is won or lost.
top_k_retrieved_chunks
Type: integer, range 1-200, default 100
Number of chunks returned by the initial hybrid search. Higher values cast a wider net but add noise and processing cost. With reranking enabled, high initial retrieval (80-100) combined with aggressive reranking (top 10-15) is the pattern that works best in production.
lexical_search_weight
Type: float, range 0-1, step 0.1, default 0.2
Weight given to BM25/lexical search in the hybrid search formula. Lexical search excels at exact keyword matching - technical terms, product codes, policy numbers, error messages. Increase this when users search for specific identifiers.
BM25 from 1994 still wins on exact term matching in my experience. Do not underestimate it.
semantic_search_weight
Type: float, range 0-1, step 0.1, default 0.8
Weight given to vector/semantic search. Finds conceptually similar content even when the exact words differ. "How do I take time off?" matches a document titled "Leave of Absence Procedure."
Balancing the weights
The two weights are independent. They do not need to sum to 1.
Common configurations:
- Semantic-heavy (0.2 / 0.8) - the default. Best for natural language questions against narrative content.
- Balanced (0.5 / 0.5) - good when content mixes prose with technical terms.
- Lexical-heavy (0.7 / 0.3) - best for codebases, API docs, error logs, or content with unique identifiers.
Start with the defaults. Adjust based on actual retrieval quality, not theory.
6. Rerank
Re-scores retrieved chunks for relevance after initial retrieval. In my experience, the most impactful quality improvement you can enable.
rerank_enabled
Type: boolean, default false
Passes retrieved chunks through a cross-encoder or LLM-based reranker. Initial retrieval optimizes for recall. Reranking optimizes for precision. All sub-fields below are only visible when enabled.
If I could change one default, it would be this. Reranking should be on.
top_k_reranked_chunks
Type: integer, range 1-200, default 15
Number of chunks to keep after reranking. Should be less than or equal to top_k_retrieved_chunks. Typical production values: 5-15 for focused answers, 15-30 for comprehensive responses.
reranker_score_filter_threshold
Type: integer, range 0-100, step 1, default 15
Minimum relevance score (percentage) a chunk must achieve to be retained. Chunks below this are discarded even if within top_k_reranked_chunks. Set to 0 to disable score-based filtering. Higher thresholds mean more precise but potentially incomplete answers.
rerank_prompt
Type: string, nullable
Custom instructions for LLM-based reranking. Customize this to prioritize certain document types, recency, or domain-specific relevance criteria. Example: "Prioritize official policy documents over meeting notes. More recent documents should rank higher."
7. Filter
Post-rerank relevance filtering using an LLM judge. The most aggressive quality gate in the pipeline.
filter_enabled
Type: boolean, default false
Each remaining chunk is evaluated individually by the LLM. Output: "Yes" or "No."
Enable for high-precision use cases where a wrong source is worse than no source. Disable when recall matters more or when latency is a concern - each chunk requires a separate LLM call. For most agents, reranking alone is enough.
filter_prompt
Type: string, nullable (visible only when filter_enabled is true)
Custom prompt for the filter LLM. The default includes five inclusion rules and five exclusion rules, plus special handling for conversational or offensive queries. Customize this to define what "relevant" means in your domain.
8. Generation
LLM output parameters for the final answer. Temperature above 0.3 for document-grounded answers is almost always wrong.
max_new_tokens
Type: integer, range 48-2048, default 2048
Maximum tokens the LLM can generate. Short-answer agents work well at 256-512. Detailed explanations need 1024-2048. Too low and responses get truncated mid-sentence.
temperature
Type: float, range 0-1, step 0.1, default 0
Controls randomness. 0 is deterministic. 1 is maximum creativity. For retrieval-based answers, keep this at 0. Higher values introduce variation that is rarely desirable when answering from documents.
The default is 0 for a reason. If your agent is citing source documents, you want the same input to produce the same output.
top_p
Type: float, range 0-1, step 0.1, default 0.9
Nucleus sampling threshold. At 0.9, the model considers the smallest set of tokens whose cumulative probability exceeds 90%. Lower values make output more focused. Adjust either temperature or top_p - not both.
frequency_penalty
Type: integer, range -2 to 2, default 0
Penalizes token repetition. Positive values reduce repeated phrases, negative values encourage them. Default of 0 applies no penalty.
random_seed
Type: integer, min 0, default 42
Seed for reproducible generation. Same seed plus temperature 0 means identical inputs produce identical outputs. Set to 0 for non-deterministic behavior.
groundedness_scores_enabled
Type: boolean, default false
Scores how grounded each response is in the retrieved sources. Useful for compliance, quality monitoring, and auditing. Turn this on for any agent where traceability matters.
avoid_commentary_enabled
Type: boolean, default false
Suppresses meta-commentary. No "Based on the documents provided..." or "Would you like me to elaborate?" Just the answer. I enable this on almost every production agent.
9. User Experience
Chat interface configuration.
suggested_queries
Type: array, nullable (entered as multi-line text, one per line)
Clickable suggestions displayed in the chat interface before the user types. Auto-generated during quick create. For manual configuration: 3-4 queries, diverse topics, natural language, max 80 characters each.
Good suggested queries teach the user what the agent is for. Bad ones waste the space.
10. Guardrails
Safety and compliance controls. For internal agents, most of these are optional. For public-facing agents, skip them at your own risk.
pii_detection_enabled
Type: boolean, default false
Scans user input for personally identifiable information (BSN numbers, IBANs) before it reaches any LLM. Detected PII is replaced with placeholders.
output_pii_filtering_enabled
Type: boolean, default false
Filters PII from the generated response before it reaches the user. Catches PII that exists in your source documents and might surface in answers.
prompt_injection_detection_enabled
Type: boolean, default false
Detects prompt injection attempts in user input - instruction overrides, encoded payloads, special character manipulation. Suspicious queries are blocked before entering the pipeline.
prompt_injection_threshold
Type: integer, range 1-10, default 3 (visible only when prompt_injection_detection_enabled is true)
Sensitivity level. Lower = more sensitive, blocks more. Higher = more permissive. Public-facing agents: 2-4. Internal teams: 6-8.
chunk_sanitization_enabled
Type: boolean, default false
Sanitizes retrieved chunks to remove potentially harmful content before passing them to the LLM for generation. Protects against poisoned documents that contain injected instructions.
11. Embed Widget
Deploy a chat widget on any external website. For full integration instructions, see the Embed Widget Setup Guide.
embed_enabled
Type: boolean, default false
Activates the embed widget for this agent. On first enable, the system generates a unique embed_token used for widget authentication.
embed_allowed_origins
Type: array, nullable (entered as multi-line text, one URL per line)
Allowlist of domains permitted to embed the widget. When null or empty, all origins are blocked - deny by default. Enter full origin URLs: https://example.com, https://docs.example.com.
Embed Snippet
Not a stored field. The editor shows a copy-paste JavaScript snippet once embed is enabled and saved. You can rotate the token from this panel if compromised.
12. Auto-Configuration Pipeline
Automated agent setup from knowledge base content.
Triggered during quick create or via artisan command. Status transitions from "configuring" to "ready" on completion. A broadcast event notifies the frontend.
Pipeline Steps
SelectRepresentativeContent - Samples 30 random chunks (max 500 characters each) from linked knowledge bases. This gives the LLM a representative slice of the content without processing everything.
GenerateDescription - The LLM generates a 1-3 sentence description (max 200 characters) from the sampled content.
GenerateSuggestedQueries - The LLM generates 4 diverse, natural-language questions (max 80 characters each) based on the actual content.
Fields affected
The pipeline overwrites description, suggested_queries, and sets status to "ready." All other fields retain their defaults.
When to use it
Quick create for rapid prototyping. For production agents, run quick create first, then refine system prompts, search weights, and guardrails based on real user interactions. The auto-generated settings are a starting point, not a destination.
13. Configuration Recipes
These are configurations from real deployments. Copy what fits, adjust from there.
High-Precision Compliance Agent
This is the configuration I use for government-facing agents. Every answer must be traceable, every source verifiable.
| Setting | Value |
|---|---|
rerank_enabled |
true |
filter_enabled |
true |
temperature |
0 |
top_k_retrieved_chunks |
100 |
top_k_reranked_chunks |
10 |
reranker_score_filter_threshold |
30 |
pii_detection_enabled |
true |
output_pii_filtering_enabled |
true |
groundedness_scores_enabled |
true |
avoid_commentary_enabled |
true |
Aggressive reranking and filtering. Zero temperature. PII protection on both input and output. Groundedness scoring for audit trails. The filter adds latency, but for compliance use cases, precision beats speed every time.
Conversational Support Agent
For internal teams that ask follow-up questions in natural conversation. Think HR helpdesk or onboarding assistant.
| Setting | Value |
|---|---|
multi_turn_enabled |
true |
retrieval_check_enabled |
true |
query_expansion_enabled |
true |
temperature |
0.2 |
max_new_tokens |
1024 |
top_p |
0.9 |
semantic_search_weight |
0.8 |
lexical_search_weight |
0.2 |
Multi-turn context for coherent conversations. Query expansion catches different phrasings. Slight temperature for natural-sounding responses. 0.2 is the highest I would go here.
Technical Documentation Agent
For codebases, API docs, and content where exact identifiers matter more than semantic similarity.
| Setting | Value |
|---|---|
lexical_search_weight |
0.7 |
semantic_search_weight |
0.3 |
query_decomposition_enabled |
true |
rerank_enabled |
true |
top_k_reranked_chunks |
15 |
temperature |
0 |
max_new_tokens |
2048 |
avoid_commentary_enabled |
true |
Lexical-heavy search finds exact function names, error codes, and configuration keys. Decomposition handles compound questions. High token limit for code examples. Zero temperature - nobody wants creative answers about API parameters.
Public-Facing Embed Agent
For customer support widgets on external websites. Exposed to untrusted users - lock it down.
| Setting | Value |
|---|---|
embed_enabled |
true |
embed_allowed_origins |
["https://yoursite.com"] |
pii_detection_enabled |
true |
prompt_injection_detection_enabled |
true |
prompt_injection_threshold |
3 |
chunk_sanitization_enabled |
true |
output_pii_filtering_enabled |
true |
retrieval_check_enabled |
true |
avoid_commentary_enabled |
true |
Every guardrail enabled. Strict origin validation. Low injection threshold. If you are embedding an agent on a public website, you cannot afford to skip any of these.
Multi-Step Agentic Assistant
For workflows that require external tool use via MCP. This is the newest pattern and the most experimental.
| Setting | Value |
|---|---|
mcp_servers |
["your-server"] |
max_tool_steps |
8 |
multi_turn_enabled |
true |
temperature |
0.1 |
max_new_tokens |
2048 |
retrieval_check_enabled |
true |
MCP servers enable tool calls to external systems. High step count for complex chains. Multi-turn keeps context across tool results. Requires Anthropic API models - not AWS Bedrock.
14. Field Reference Table
| Field | Type | Range / Constraints | Default | Section |
|---|---|---|---|---|
name |
string | required | - | General |
description |
string | nullable | null | General |
knowledgebases |
array | knowledge base UUIDs | [] | General |
mcp_servers |
array | configured server names | null | General |
max_tool_steps |
integer | 1-10 | 10 | General |
system_prompt |
string | required | (built-in) | System Prompts |
no_retrieval_prompt |
string | required | (built-in) | System Prompts |
multi_turn_enabled |
boolean | - | true | Query Understanding |
multi_turn_system_prompt |
string | required when multi-turn on | (built-in) | Query Understanding |
retrieval_check_enabled |
boolean | - | true | Query Understanding |
query_expansion_enabled |
boolean | - | false | Query Reformulation |
query_expansion_instructions |
string | nullable | null | Query Reformulation |
query_expansion_examples |
array | nullable, one per line | null | Query Reformulation |
query_decomposition_enabled |
boolean | - | false | Query Reformulation |
query_decomposition_examples |
array | nullable, one per line | null | Query Reformulation |
top_k_retrieved_chunks |
integer | 1-200 | 100 | Retrieval |
lexical_search_weight |
float | 0-1, step 0.1 | 0.2 | Retrieval |
semantic_search_weight |
float | 0-1, step 0.1 | 0.8 | Retrieval |
rerank_enabled |
boolean | - | false | Rerank |
top_k_reranked_chunks |
integer | 1-200 | 15 | Rerank |
reranker_score_filter_threshold |
integer | 0-100 | 15 | Rerank |
rerank_prompt |
string | nullable | null | Rerank |
filter_enabled |
boolean | - | false | Filter |
filter_prompt |
string | nullable | null | Filter |
max_new_tokens |
integer | 48-2048 | 2048 | Generation |
temperature |
float | 0-1, step 0.1 | 0 | Generation |
top_p |
float | 0-1, step 0.1 | 0.9 | Generation |
frequency_penalty |
integer | -2 to 2 | 0 | Generation |
random_seed |
integer | min 0 | 42 | Generation |
groundedness_scores_enabled |
boolean | - | false | Generation |
avoid_commentary_enabled |
boolean | - | false | Generation |
suggested_queries |
array | nullable, one per line | null | User Experience |
pii_detection_enabled |
boolean | - | false | Guardrails |
output_pii_filtering_enabled |
boolean | - | false | Guardrails |
prompt_injection_detection_enabled |
boolean | - | false | Guardrails |
prompt_injection_threshold |
integer | 1-10 | 3 | Guardrails |
chunk_sanitization_enabled |
boolean | - | false | Guardrails |
embed_enabled |
boolean | - | false | Embed |
embed_allowed_origins |
array | nullable, URL per line | null | Embed |
Related Resources
- How Our RAG Pipeline Works - technical deep-dive into the 16-step pipeline
- Embed Widget Setup Guide - full embed integration instructions
- AI Agents - feature overview
- API Reference - programmatic agent management
- Security - full security architecture details