Document Processing Pipeline
How Knowably transforms PDFs, Office documents, Confluence pages, and more into searchable, vector-indexed chunks ready for AI retrieval.
What happens between the moment a document enters Knowably and the moment it becomes searchable?
More than you would expect. A SharePoint PDF, a Confluence page, a Jira ticket, and an uploaded CSV all need to end up in the same place: a vector-indexed chunk store where an AI agent can retrieve them in milliseconds. Getting there requires format detection, markdown conversion, intelligent splitting, and embedding generation - across documents that range from 2KB plain text files to 200MB slide decks.
This page is the technical companion to our How It Works overview. It covers the full ingestion pipeline. For the retrieval side - what happens when a user asks a question - see How Our RAG Pipeline Works.
The Job Chain
I split the pipeline into discrete queued jobs because I wanted granular retry. If extraction fails, the download is still good. If embedding fails on chunk 47 of a 200-chunk document, only that chunk retries. The other 199 are already indexed and searchable.
Each stage runs as an independent Laravel job implementing ShouldQueue. Each stage dispatches the next upon success.
The chain looks like this:
DownloadDocumentContentJob - ExtractAndSplitDocumentJob - IndexChunkJob (one per chunk) - CheckDocumentIngestionStatusJob - CheckDataSourceIngestionStatusJob
The chain runs on Laravel's queue system. Throughput scales horizontally by adding queue workers. The embedding stage - typically the bottleneck - can run on dedicated workers while download and extraction share another pool.
Data Source Client Abstraction
Every data source type has its own client implementation. SharePoint, Confluence, Jira, Email, GitHub, S3, Google Drive, Websites, and file uploads - each has a dedicated client class that knows how to talk to its source system.
All clients extend AbstractClient, which provides default implementations for extraction, splitting, and indexing. Individual clients override only what they need to. A Confluence client receives HTML directly from the Confluence API - no binary download step. A SharePoint client needs OAuth token refresh and binary file download. The abstraction handles this cleanly.
The DataSourceType enum resolves the correct client and config classes at runtime through a factory pattern. When a document needs processing, the pipeline calls $document->dataSource->client(), and the enum returns the right implementation. No conditionals in the pipeline code. No if-else chains checking source types.
The DataSourceClient contract defines the full interface: collectDocuments(), getDocumentContent(), extractDocumentContent(), splitDocument(), and indexChunk(). Every client implements these methods, either directly or through the abstract base class.
Three Microservices
The compute-intensive operations run in dedicated Python microservices, each accessible over HTTP. I chose Python for these because PHP is not great at parsing PDFs with embedded tables. Or running ML models. The right tool for the job.
ToMarkdown converts binary documents to clean markdown. It accepts file content and a filename via multipart upload, and returns normalized markdown text. PDFs, Word documents, Excel spreadsheets, PowerPoint presentations, a dozen other formats - all handled here.
TextSplitter takes markdown content and splits it into semantically meaningful chunks. It uses the semantic-text-splitter Python library under the hood, with configurable parameters for chunk size, overlap, and splitter type.
Embedding generates vector embeddings from text content. It accepts a text string and returns a dense vector array that captures the semantic meaning of the input.
Each service has a 600-second timeout and a 15-second connection timeout. Generous limits that accommodate large documents without blocking queue workers. The services are decoupled behind contracts - ConvertsToMarkdown, SplitsText, and GeneratesIndex - so implementations can be swapped without touching pipeline code.
Stage 1: Collected
When a data source syncs, its client discovers documents in the remote system and creates Document records in the database. Each record stores an external_id, name, extension, size, mime_type, and a metadata JSON field with source-specific data like webLink, web_url, or links.webui.
The document's initial status is collected. A remote reference exists, but no content has been downloaded yet.
Collection is triggered by initial data source configuration, a manual re-sync from the UI, or an API call. The DownloadDataSourceDocumentsJob iterates all collected documents using cursor-based iteration - preventing memory exhaustion on large data sources - and dispatches individual download jobs for each one.
Download jobs use Laravel's RateLimited middleware with a knowledgebase-download limiter. SharePoint, Confluence, and other cloud APIs throttle aggressively. Without rate limiting, a large sync hits 429 responses within seconds.
Stage 2: Downloaded
DownloadDocumentContentJob fetches raw content from the external system. It is configured with 3 retry attempts and exponential backoff at 10 and 60 seconds.
The client's getDocumentContent() method does the actual fetching. Content goes into the document's content field. Binary content from certain APIs - PDFs from SharePoint, for example - may arrive base64-encoded. The pipeline's extractDocumentContent() method auto-detects this using strict base64 validation and decodes transparently.
Error handling is deliberate here.
Client errors (HTTP 4xx) mark the document as invalid with the HTTP status code as the reason. A 404 means the document was deleted from the source. A 410 means it was permanently removed. These are not retried - permanently unavailable resources.
Server errors (HTTP 5xx) re-throw the exception, letting the job retry automatically through Laravel's retry mechanism. Transient failures resolve themselves.
Empty content is handled gracefully. Documents with no extractable content skip the extraction pipeline entirely and move to ready status.
If the document has content, the job dispatches ExtractAndSplitDocumentJob to continue the chain.
Stage 3: Extracted
Extraction converts raw document content into clean markdown. This is where format diversity collapses into a single representation.
Format Detection
ParserFactory selects the appropriate parser based on file extension. The default path routes through GenericToMarkdownParser, which delegates to the ToMarkdown microservice. For content that is already text-based - Confluence pages (HTML), Jira issues (structured text) - the CopyContentParser acts as a pass-through. No conversion needed.
The parser contract is minimal: parse(string $content, string $fileName): string. Content in, markdown out.
The ToMarkdown Microservice
The ToMarkdown microservice exists because PHP is not great at parsing PDFs with embedded tables. Or extracting text from scanned images. Python has better libraries for this, and wrapping them in an HTTP service keeps the PHP side clean.
Supported input formats include PDF (with OCR for scanned pages), Word (DOC, DOCX), Excel (XLS, XLSX), PowerPoint (PPT, PPTX), RTF, OpenDocument formats, CSV, HTML, and plain text. Output is normalized markdown that preserves heading hierarchy, lists, tables, and code blocks.
Ingestion Modes
Two modes are available, configurable per data source.
Fast mode is the default. Optimized for text-based documents, it processes significantly faster than Hi-Res on text-heavy content. Images within documents are still processed in Hi-Res automatically - you do not sacrifice visual content for speed.
Hi-Res mode extracts text, images, and tables from complex layouts in PDFs, Word documents, and PowerPoint files. Designed for documents where spatial layout carries meaning - scanned forms, multi-column reports, slide decks with embedded diagrams. Unsupported formats fall back to Fast mode automatically.
Advanced Extraction Features
Three optional features give you more control over extraction.
Figure captioning uses AI to generate captions for embedded images. This makes visual content searchable. Custom captioning prompts are supported, so you can tailor descriptions to your domain. An engineering diagram gets a different caption than a marketing infographic.
Table splitting breaks large tables into smaller, more digestible chunks. The threshold is configurable via max_split_table_cells. Without this, a 500-row table becomes a single unwieldy chunk that dilutes retrieval precision.
Hierarchy-based contextualization preserves the parent heading chain when splitting nested document sections. A chunk that reads "Must be completed within 30 days" becomes much more useful when it carries the context "Employee Handbook > Leave Policy > Annual Leave > Must be completed within 30 days."
Stage 4: Chunked
Chunking strategy matters more than embedding model choice. I have seen retrieval quality change dramatically just by switching from static-length to markdown-aware splitting. A heading and its content belong together. A code function should not be split mid-block. These are not theoretical concerns - they directly affect whether your agent returns a useful answer.
The TextSplitter Microservice
A Python Flask service powered by the semantic-text-splitter library. It offers three splitter types.
MarkdownSplitter is the default, and my recommendation for most content. It respects markdown structure - headings, code blocks, lists - as semantic boundaries. A heading and its content stay together rather than being split mid-paragraph.
TextSplitter handles generic unstructured content. It splits at sentence and paragraph boundaries without markdown awareness.
CodeSplitter is optimized for source code. It respects function and class boundaries so code snippets remain coherent.
All splitters accept four configurable parameters: max_characters (default 1000), overlap (default 200 characters between adjacent chunks), trim (whitespace trimming, default true), and type (splitter selection). These defaults work well for most content. For specialized use cases - legal documents, codebases, research papers - you adjust per data source.
Chunking Modes
Beyond the splitter type, the chunking mode controls strategy at the data source level.
Hierarchy Depth (default) splits based on document structure depth levels. Hierarchy Heading splits on heading boundaries, keeping heading context with the content below. Static Length splits at fixed token lengths with configurable min and max via max_chunk_length_tokens and min_chunk_length_tokens. Page Level splits per page, useful for PDFs where page boundaries are meaningful.
Chunk Storage
Chunks land in the knowledgebases_chunks table. Each chunk belongs to a document via a document_id foreign key. The firstOrCreate pattern on chunk content prevents duplicate chunks when a document is re-processed - idempotency built in.
Each chunk record stores: text content, metadata (JSON), embeddings_768, embeddings_1024, and indexed_at.
The dual embedding columns deserve explanation. I support both 768 and 1024 dimensions because embedding models evolve. You should be able to switch models without running a migration. Deploy a model that outputs 1024-dimensional vectors, and the pipeline uses the right column automatically. No schema change, no downtime, no re-deployment.
Stage 5: Indexed
Each chunk needs a vector embedding before it is searchable. The IndexChunkJob handles this - one job per chunk, so embedding generation parallelizes across as many queue workers as you have available.
The job is configured with 2 retry attempts and 10-second backoff. It is idempotent: chunks that already have an indexed_at timestamp are skipped. Re-running is safe.
The embedding service returns a dense vector array. The pipeline dynamically selects embeddings_768 or embeddings_1024 based on the returned array length.
Vector Storage in pgvector
I store embeddings directly in PostgreSQL using the pgvector extension. No separate vector database. This is a deliberate choice. One database for relational data and vector data means simpler deployments, simpler backups, and simpler operations. I have not hit a scale where a dedicated vector store would outperform pgvector with proper indexing.
pgvector supports both cosine similarity and inner product distance calculations. At query time, approximate nearest neighbor (ANN) indexes enable fast vector search without scanning every row.
After each chunk is indexed, the job dispatches CheckDocumentIngestionStatusJob to see if the parent document is complete.
Stage 6: Ready
Document Completion
CheckDocumentIngestionStatusJob runs after each chunk finishes indexing. It checks whether all chunks for the document have a non-null indexed_at timestamp. When they do, the document status moves to ready and the ready_at timestamp is recorded.
A DocumentStatusUpdated event fires, broadcasting immediately over WebSocket on a private documents channel. The payload includes the document's UUID, name, extension, size, mime type, public status, and status reason. The frontend receives this in real-time and updates the document table without a page refresh.
Data Source Completion
CheckDataSourceIngestionStatusJob runs after each document completes. It checks whether all documents in the data source have a non-null ready_at. When they do, the data source status moves to ready.
Document Status Lifecycle
Internal Statuses
A document moves through six statuses: collected, downloaded, extracted, chunked, indexed, ready.
Two terminal failure states exist. Invalid is permanent - the source returned a 404 or 410. The document will not be retried unless manually re-synced. Failed is transient - a timeout, a parsing error, a temporary service outage. The error message is stored in status_reason and reported to Bugsnag.
Public Status Mapping
Internal statuses are mapped to simplified public statuses for the UI. collected becomes pending. Everything between downloaded and indexed becomes processing. ready becomes processed. invalid and failed pass through unchanged.
Users see three states that matter: waiting, working, done. Plus two error states when something goes wrong.
Data Source Statuses
Data sources have their own lifecycle: configuring, collecting, importing, indexing, ready, and failed. Status counts per data source - total documents, processing, processed, failed, total size - provide a high-level overview without drilling into individual documents.
Error Handling and Retries
Each stage has its own retry configuration tailored to the failure modes it encounters.
DownloadDocumentContentJob: 3 attempts, backoff at 10s and 60s. External APIs are flaky. Three tries with increasing delays handles most transient failures.
ExtractAndSplitDocumentJob: 2 attempts, 10s backoff. Parsing failures are usually deterministic - if a file cannot be parsed, retrying rarely helps. Two tries catches transient microservice issues.
IndexChunkJob: 2 attempts, 10s backoff. The embedding service is generally stable, but network hiccups happen.
When all retries are exhausted, the failed() method on the job fires. It records the error in status_reason, updates the document status to failed, dispatches a status update event for the UI, and reports to Bugsnag. Nothing is silently swallowed.
Batch Operations
Document Batch Sync
The API accepts an array of document UUIDs. Each document is authorized individually - no blanket permissions. Each one resets to collected status, clears its ready_at and status_reason, and re-enters the pipeline from the top.
Data Source Sync
A full data source re-sync re-discovers documents from the remote system. It detects new documents, updated content, and removed files. Only changed content is reprocessed - incremental by default.
Media Management
Uploaded files are managed through Spatie Media Library. The media model is tenant-aware, linking each record to its parent data source. For remote sources, the original URL is preserved in document metadata under keys like webLink, web_url, links.webui, or url. The document model resolves the correct external link regardless of source system.
Performance and Scaling
Horizontal scaling. Every pipeline stage runs as a queued job. Add workers to increase throughput. The embedding stage can have dedicated workers while other stages share a pool.
Memory efficiency. Cursor-based iteration on large data sources prevents loading thousands of document records into memory at once.
Idempotency. Re-running any stage on already-processed data is safe. IndexChunkJob skips chunks with an indexed_at timestamp. firstOrCreate on chunks prevents duplicates. Batch re-sync resets status and re-enters the pipeline cleanly.
Rate limiting. External API calls go through Laravel's rate limiter to prevent throttling by source systems. Better to process slowly than to get blocked.
Generous timeouts. The microservices allow 600 seconds per request. A 100-page PDF with OCR takes time. The pipeline waits rather than failing on legitimate workloads.
Observability
The full ingestion pipeline is observable through multiple channels: document and data source status in the admin UI, real-time WebSocket status updates, failed job tracking with error reasons, and Bugsnag error reporting for unhandled failures. For retrieval pipeline tracing, I use Langfuse - covered separately in the observability documentation.
What This Means in Practice
Connect a SharePoint site with 10,000 files in mixed formats. The pipeline discovers them, downloads them respecting API rate limits, converts them all to markdown, splits them into focused passages, generates vector embeddings, and makes them searchable - with real-time status updates the entire way.
Every stage is independently retryable. Every stage is horizontally scalable. Every failure is tracked and reported.
You configure your data sources and agents. I handle the pipeline.
Explore data source integrations | See how retrieval works | Get Started