Understand how chunking strategies and embedding models affect retrieval quality
Chunking and vectorization are performed automatically during data import. Chunking determines the smallest unit for retrieval, while vectorization determines the precision of semantic matching. Both are configured in the index settings step of Importing Data and cannot be modified after creation.
The following chunking methods are available when importing data:
Maximum Chunk Length: Default 600; range 10–6000 tokens.
After chunking, the system uses an embedding model to convert text chunks into vectors and writes them to the index. The embedding model is selected when creating the knowledge base and cannot be changed afterward.
During data import, the cost details panel on the right displays the currently used embedding model and reranking model.
Once chunking is complete, view and manage chunks in the Chunk Details tab on the knowledge base detail page. For more information, see Document Management — Chunk Management.
Chunk-related APIs:
Chunking Methods
The following chunking methods are available when importing data:
| Chunking Method | Description | Use Cases |
|---|---|---|
| Smart Chunking (default) | Automatically detects section and paragraph boundaries | Most scenarios |
| Fixed-Length Chunking | Splits text into equal-length chunks based on a fixed token count | Scenarios with strict token-count requirements |
| Page-Based Chunking | Splits by PDF page boundaries | Documents where each page conveys an independent topic |
| Heading-Based Chunking | Splits by Markdown / HTML heading levels | Documents where headings demarcate independent topics |
| Regex-Based Chunking | Splits at positions matched by a custom regular expression | Text with consistent delimiters |
| Delimiter-Based Chunking | Splits using specified delimiter characters | Documents where content is separated by specific identifiers |
Chunking Selection Guidelines
| Scenario | Recommended Configuration | Reason |
|---|---|---|
| FAQ Documents | Fixed-Length Chunking, 256 tokens | Q&A pairs are short and self-contained |
| Product Manuals | Smart Chunking, 600 tokens (default) | Clear chapter structure |
| Long Reports | Smart Chunking, 2048 tokens | Preserves more contextual information |
| Legal Contracts | Heading-Based Chunking, 512 tokens | Clauses are clearly organized by heading hierarchy |