Skip to main content
Cookbook

RAG Performance Optimization

Diagnose issues stage-by-stage across the RAG pipeline and apply targeted improvements to systematically enhance retrieval and response quality

RAG (Retrieval-Augmented Generation) performance is determined by three sequential stages:
  1. Indexing: Document parsing, chunking, and vectorization
  2. Retrieval: Matching and retrieving relevant chunks from the index based on a query
  3. Generation: The LLM generating the final answer using both the retrieved content and the query
Each stage may introduce performance bottlenecks. This guide follows a diagnose → resolve pattern to help you identify root causes and apply corresponding optimization strategies.

Step 1: Establish an Evaluation Baseline

Before optimizing, define a quantifiable evaluation framework—otherwise, you cannot objectively assess whether improvements are effective.

Build an Evaluation Dataset

Prepare at least 100 test cases, each containing a question and its expected answer. Ensure coverage across the following types:
TypeExample
Factual"What is the warranty period for Product X?"
Comparative"Compare the key differences between Product X and Product Y."
Tutorial"How do I install Product X?"
Analytical"Why has Product X’s sales increased over the past three months?"

Run the Baseline Test

Execute the full evaluation set once using either the Q&A service’s debugging console or the chat API. Record both the retrieved chunks and the generated responses for every test case. This report serves as your reference baseline for all subsequent optimizations.

Step 2: Diagnosis and Remediation

Analyze underperforming test cases in the baseline report one by one, categorize the issue, and apply the appropriate solution below.

2.1 Failed Retrieval: Relevant Knowledge Exists but Is Not Retrieved

The knowledge exists in the knowledge base, yet the retrieval system fails to match it.
If domain knowledge is missing from the knowledge base, the LLM cannot answer related questions. Verify whether the evaluation set covers topics already present in your knowledge base—and promptly add any missing content.
Poor document formatting can cause critical information to be lost during parsing. We recommend:
  • Using clear, hierarchical headings and well-structured content
  • Removing watermarks
  • Avoiding complex tables with merged or cross-page cells
  • Preferring Markdown format; for PDF/DOCX files, convert them to Markdown before ingestion
If queries are predominantly in English or another language, ensure source documents use the same language. For technical terms, consider multilingual handling (e.g., bilingual annotations like “ML / Machine Learning / 机器学习”).
Standardize references to the same entity. For example, unify “ML”, “Machine Learning”, and “机器学习” into a single canonical form such as “机器学习”.
You can feed documents to an LLM to assist with terminology standardization. For long documents, split them into smaller segments first and process each segment individually.
In multi-turn conversations, user inputs may be incomplete or omit contextual information. Enabling Query Rewriting allows the system to automatically expand or rephrase queries using conversation history—improving retrieval relevance.Example: User input phone X1 → rewritten as Please provide all currently available versions of phone X1 and their specifications from the product catalog.Enable the Query Rewriting toggle in the knowledge base-specific configuration section of either Knowledge Retrieval or Knowledge Q&A.

2.2 Failed Retrieval: Irrelevant Chunks Are Retrieved

Relevant knowledge exists in the knowledge base, but irrelevant chunks are retrieved instead.
Typical Issue: Your knowledge base contains documents from multiple categories (e.g., A and B), yet queries intended for Category A return results from Category B.Solution: Assign tags to documents and filter retrieval results by tag to narrow scope.Setup Options:
  • During Import: Add tags in Step 3 of Data Upload
  • After Import: Click “More” → “Tags” in the document list
Usage Options:
  • Configure Tag Filtering in the knowledge base-specific settings of Knowledge Retrieval or Knowledge Q&A
  • Specify tags in the request payload when calling via API
Typical Issue: Multiple documents share similar structure (e.g., all contain a “Feature Overview” section), causing retrieval of wrong-document content.Solution: Define metadata fields for documents. The system performs structured pre-filtering before vector search to precisely locate target documents.How It Works:
  1. Extract metadata from the query (e.g., {"key": "name", "value": "phone X1"})
  2. Filter chunks that contain matching metadata values
  3. Perform vector search only on the filtered subset
Impact Comparison: Without metadata, irrelevant chunks (similarity score 40–41%) may rank ahead of the correct chunk (37%). With metadata enabled, the correct chunk’s similarity rises to 65% and ranks #1.View and configure metadata in the document list by clicking “Meta Info”. Common use cases include embedding timestamps to filter for recent content.

2.3 Suboptimal Chunking

Inappropriate chunking leads to the following issues:
IssueManifestationImpact
Chunks too shortEach chunk contains only one sentence, lacking contextRetrieval fails to match semantically relevant content
Chunks too longA single chunk mixes multiple topicsRetrieval returns excessive irrelevant content
Semantic truncationChunks are forcibly cut mid-sentenceRetrieved content is incomplete or malformed
Smart chunking dynamically selects boundaries based on semantic coherence—not fixed-length splitting:
  1. First, the system splits documents into paragraphs using built-in sentence delimiters
  2. Then, it determines optimal chunk boundaries based on inter-paragraph semantic similarity
Recommended for most use cases. Select this option when creating a knowledge base or importing data.
Automated chunking may still produce unexpected truncations or parsing errors (e.g., spaces rendered as %20). After import, manually spot-check chunk quality.Navigate to the Chunk Details tab in the knowledge base detail page to view all chunks. Edit and correct problematic ones directly—the updated chunks take effect immediately for retrieval.
Corrections affect only indexed chunks—not original source documents. Re-imported documents require re-inspection.
For edge cases where auto-chunking falls short, perform fine-grained adjustments at the chunk level:
  • Add Chunk: Insert missing critical information or additional context. Must be associated with an existing document; max length: 6000 characters
  • Delete Chunk: Remove erroneous or duplicate chunks—other chunks in the same document remain unaffected
See Document Management — Chunk Management for instructions.

2.4 Poor Re-Ranking

Relevant chunks are retrieved—but poorly ranked, resulting in suboptimal final output.
The similarity threshold filters out chunks scoring below the specified value after ranking.
  • Too high: May discard all relevant chunks, leaving the LLM under-informed
  • Too low: May retrieve many low-relevance chunks, introducing noise
There is no universal optimal threshold—tune iteratively using the debugging console while observing retrieval outputs. Start with 0.2.Configuration location: Similarity Threshold (range: 0.01–1.0) in the knowledge base-specific settings of Knowledge Retrieval or Knowledge Q&A.
Controls how many top-ranked chunks are returned. For complex tasks requiring summarization, enumeration, or comparison, increasing the count (e.g., K=20) often yields better results than limiting to just 5.However, larger counts aren’t always better—excessive chunks may exceed the LLM’s input token limit, leading to truncation.Configuration location: Maximum Retrieval Count (range: 1–20) in knowledge base-specific settings.
The ranking model re-orders initially retrieved chunks to determine the final sequence passed to the LLM. Available modes:
ModeUse Case
Q&A Mode (default)User asks a complete question expecting direct answers from chunks
Similarity ModeQuery and chunk phrasing styles align closely (e.g., searching historical records or policy clauses)
Custom Advanced ModeSpecialized ranking logic required (e.g., “prioritize content from the last 12 months”)
  • Per-Knowledge-Base Configuration: Expand the knowledge base panel in retrieval or Q&A services to select model and mode
  • Global Reranking Model: Configure once in Knowledge Retrieval to apply uniformly across all bound knowledge bases

2.5 LLM Misunderstanding

Correct chunks are retrieved—but the LLM generates inaccurate or incomplete answers.
Typical Issue: Output appears mechanically stitched together; the model fails to grasp relationships between query and retrieved knowledge.LLMs differ significantly in instruction-following ability, factual comprehension, and long-context handling:
ScenarioRecommended Model
Simple fact lookup & summarizationSmaller-parameter models (e.g., qwen3-flash)
Complex logical reasoningLarger-parameter models (e.g., qwen3-max)
Heavy reliance on many retrieved chunksModels with longer context windows
Change the Model in the Q&A service configuration page.
Typical Issue: Output violates formatting requirements or lacks completeness.Three proven techniques:Technique 1: Constrain Output Scope
If the provided information is insufficient to answer the question, explicitly state:
"Based on current information, I cannot answer this question." Never fabricate answers.
Technique 2: Include Few-Shot ExamplesEmbed desired input-output examples in the prompt to guide format and style.
# Instruction
Extract technical specifications from the text below and output JSON using the field format shown in the example.

# Example
## Input: Stardust S9Pro, 6.9-inch display, 512GB storage, 16GB RAM, 6000mAh battery
## Output: {"product":"Stardust S9 Pro","screen_size":"6.9inch","ram_size":"16GB","battery":"6000mAh"}
Technique 3: Delimit Knowledge ContentClearly separate instructions from knowledge content using explicit delimiters (e.g., ---, <|context|>) to improve LLM structural understanding.
Typical Issue: Answers incorporate the LLM’s general knowledge rather than strictly adhering to the knowledge base.Turn on Refusal in the Q&A service configuration. When enabled, the model refuses to answer—and returns a custom message—if retrieval results are insufficient, instead of hallucinating.Configuration location: Refusal toggle in Knowledge Q&A under Generation Control Parameters.
Typical Issue: Identical queries yield inconsistent outputs—or distinct queries produce identical answers.Adjust the temperature parameter via the model configuration button:
  • Higher values: Increase output diversity—ideal for creative writing or brainstorming
  • Lower values: Increase output determinism—ideal for factual queries or technical documentation
In the Q&A service config page, click the Settings icon next to the model name to adjust.

2.6 Multi-Knowledge-Base Optimization

When binding multiple knowledge bases, apply these strategies to improve joint retrieval effectiveness:
StrategyDescription
Set Knowledge Base Weight/PriorityAssign higher weights or priority to core knowledge bases so their results dominate final ranking
Enable Knowledge Base RoutingSystem automatically routes queries to relevant knowledge bases based on intent—reducing noise from irrelevant sources. Configure in Knowledge Retrieval
Choose Reranking ModeQ&A mode suits QA scenarios; Similarity mode fits semantic paragraph search; Custom Advanced mode supports specialized sorting logic

Step 3: Continuous Iteration

After each configuration change:
  1. Re-run Evaluation: Execute the same evaluation set to quantify improvement
  2. Compare & Analyze: Contrast results against the baseline—confirm which issues were resolved and whether new regressions emerged
  3. Expand Test Coverage: Add newly discovered production failures (“bad cases”) to your evaluation set to broaden coverage
Performance optimization is iterative—typically requiring 2–3 rounds. Prioritize indexing and retrieval improvements before tuning generation behavior.
Getting Started
Playground
Data Ingestion
Knowledge Service
Cookbook
Integration
Settings, Billing & Limits
Changelog