Dashboard and Analytics
Monitor costs, usage patterns, and agent performance from a single dashboard. Track token spend, query volume, and user engagement across all agents.
I built the dashboard because I needed to answer one question: how much is this costing?
When you run AI agents against live data sources, every query burns tokens. Every token has a price. And that price varies by provider, by model, by whether it is input or output. Without a single screen that rolls all of this up, you are guessing at your budget.
Knowably's dashboard gives workspace administrators one view with everything that matters: agent activity, query volume, token consumption, storage usage, and cost. It auto-refreshes every 30 seconds. No configuration, no tracking code. Use the platform, and the data appears.
The Agent That Was Burning Money
A few weeks after launching the first production workspace, I noticed one agent was consuming three times the tokens of every other agent combined. Same model. Similar query volume. Wildly different cost.
I opened the dashboard. The input/output token split told the story immediately. Output tokens were normal. Input tokens were through the roof. This agent had 15 retrieved chunks configured with reranking enabled - each query was stuffing thousands of tokens of document context into every prompt before the model even started generating.
I dropped the chunk count from 15 to 8 and tightened the reranking threshold. Token cost fell by 40% overnight. Response quality actually improved because the model was no longer drowning in marginally relevant content.
Without the dashboard, I would have found this in the invoice. A month later.
That is why every metric on this screen exists. Not because dashboards are nice to have, but because I have been burned by the alternative.
What You See When You Open It
Six metric cards across the top. Agents, data sources, queries this month, tokens consumed, storage used, and credits spent. Each one links to its management page.
The credits number is the one your finance team will ask about. It is calculated from actual token counts and per-model pricing - input and output tracked separately because they cost different amounts. The rest gives you the operational picture at a glance.
Below the cards, three sections: trends, token breakdown, and performance. All covering a rolling 30-day window.
Token Tracking - Where the Money Goes
Every LLM interaction gets its own record. Provider. Model. Operation type. Input tokens. Output tokens. Calculated cost. Tied to the user who triggered it.
If someone on your team is running expensive queries, you will know.
The Input/Output Split
This split matters more than most people realize.
High input tokens means large system prompts or too many retrieved chunks. High output tokens means verbose responses. Different problems, different solutions. The dashboard shows you which one you are dealing with.
That agent I mentioned earlier? Pure input problem. I would have spent hours guessing without this breakdown.
Provider Distribution
A 30-day view breaks down token usage by provider. See whether OpenAI or Anthropic consumes the bulk of your tokens. Drill down to model-level usage.
This is where model switching decisions get concrete. I moved one workspace from GPT-4o to Claude Sonnet for a batch of straightforward Q&A agents. Same answer quality. Noticeably lower cost. The provider distribution chart made the before-and-after obvious.
Query Trends and Performance
Every query through the RAG pipeline is logged: which agent handled it, which user asked, response time in milliseconds, chunk count, tokens consumed, success or failure.
The 30-day trend plots daily query distribution. You spot patterns. Weekday peaks. Quiet weekends. A spike after a new data source connects. A sudden drop that means users stopped trusting an agent.
Average response time tracks latency for successful queries. If this number creeps up, something in your pipeline needs attention - slow retrieval, an overloaded model, or too many reranking passes.
Success rate should sit above 95%. Below that, check the error type distribution. Timeouts? Data source errors? Prompt injection detection blocking legitimate questions? The breakdown tells you where to look. A slow agent is a tuning issue. A failing data source is an integration issue. The dashboard makes the distinction clear.
Storage usage plots the last three months of ingested document content - text from PDFs, SharePoint files, Confluence pages, everything your data sources pull in.
Per-Agent Statistics
Not all agents are equal. Some handle hundreds of queries a day. Others sit idle.
The dashboard ranks the top five agents by query volume so you see which ones deliver value. For each agent: response time, chunk count, token usage per query. You identify which agents are expensive and which are lean.
This drives practical decisions. An agent with three queries per month is a candidate for retirement. An agent with high token costs might benefit from a cheaper model or fewer retrieved chunks. An agent with slow response times might need stricter reranking thresholds.
The link between dashboard metrics and agent configuration is deliberate. See the numbers, then adjust the settings. That feedback loop is the whole point.
What This Means in Practice
The dashboard is a management tool, not a vanity screen.
Finance wants to know what AI costs. The dashboard answers in dollars. IT wants to know if agents are performing. The dashboard answers in milliseconds and success rates. Product wants to know if users are engaged. The dashboard answers in daily active users and query volume.
Everything is collected automatically. No SDK, no tracking code, no third-party analytics service. Only workspace administrators see the dashboard - everyone else gets redirected to the agents page. Each workspace sees only its own data, enforced at the database level.
I check the dashboard every morning. Not because I have to. Because the one time I did not, it cost me a month of overspend on a misconfigured agent.