Alibaba Cloud Bailian provides an automated evaluation feature that leverages large language models (LLMs) and the application’s knowledge base to automatically generate evaluation datasets, assess agent responses, and produce evaluation reports with optimization recommendations.
Manually evaluating agent applications requires manually constructing evaluation datasets—a time-consuming and labor-intensive process. Moreover, evaluation results depend heavily on domain experts’ subjective judgments: while this ensures professional rigor, it makes the process difficult to quantify and potentially biased. Alibaba Cloud Bailian’s Automated Evaluation feature addresses these challenges by leveraging LLMs and the application’s knowledge base to automatically generate evaluation datasets, assess agent responses, and generate comprehensive evaluation reports—including scores, error analysis, and actionable optimization suggestions.
Automated Evaluation supports two modes:
A complete Automated Evaluation consists of four sequential phases:
Creating an Evaluation Task → Configuring the Evaluation Dataset → Setting Evaluation Rules → Executing Evaluation
You can configure the evaluation dataset in two ways:
The system uses an LLM to score each agent response on a 1–5 scale. Scoring compares the agent’s output against the reference answer in the evaluation dataset, assessing accuracy, completeness, and relevance. Scoring rules:
For BadCases scoring below 4, the system automatically identifies which stage of the RAG pipeline failed. Below are root-cause categories, meanings, and corresponding optimization guidance:
A single evaluation reflects application performance only at a specific point in time. To sustain high agent quality, integrate Automated Evaluation into daily development and operations workflows.
Trigger an evaluation after any of the following changes:
Follow this iterative cycle:
- Single-Application Evaluation: Deeply evaluates the performance of a single agent application, generating a detailed report containing scores, error analysis, and optimization recommendations—enabling rapid issue identification and targeted improvements.
- Cross-Application Comparative Evaluation: Evaluates multiple applications (or different versions of the same application) against a shared benchmark, enabling objective comparison across key metrics—ideal for application selection or validating version iteration effectiveness.
Prerequisites
- Automated Evaluation is only available for published agent applications.
- The evaluation dataset is automatically generated based on the application’s knowledge base—ensure your application has at least one knowledge base configured.
- Agent application evaluation relies on inference process data—ensure the Application Observability feature is enabled, and add the target application(s) to the observability list.
- RAM users (sub-accounts) must be granted either the
AdministratororApplication Evaluation - Operationpermission to use Automated Evaluation.
Workflow
A complete Automated Evaluation consists of four sequential phases:Creating an Evaluation Task → Configuring the Evaluation Dataset → Setting Evaluation Rules → Executing Evaluation
Creating an Evaluation Task
1
Navigate to the Automated Evaluation Page
Go to the Alibaba Cloud Bailian console Automated Evaluation page and click Create Evaluation Task.If Application Observability has not yet been enabled, click Go Now in the pop-up dialog to enable it immediately.
2
Select Applications to Evaluate
Select the agent application(s) you want to evaluate. You may select one application for single-app evaluation—or up to eight applications for cross-application comparative evaluation.
- Only published applications with configured knowledge bases are supported.
- Different versions of the same agent are treated as distinct, independent applications.
- For cross-application comparative evaluation, all selected applications must share at least one common knowledge base.
3
Select Knowledge Base(s)
Choose the knowledge base(s) to use for evaluation.
- For single-application evaluation: Select one or more knowledge bases from those associated with the selected application.
- For cross-application comparative evaluation: The system lists only knowledge bases common to all selected applications—choose one or more to generate the evaluation dataset.
4
Confirm and Proceed
Verify your selections, then click Next. If any selected application lacks Application Observability, click Enable Now & Proceed in the pop-up dialog.
Configuring the Evaluation Dataset
You can configure the evaluation dataset in two ways:
- Generate Evaluation Dataset: Automatically generates questions and reference answers using an LLM, based on the previously selected knowledge base(s). All generated datasets are viewable on the Evaluation Datasets page.
- Select Existing Evaluation Dataset: Reuse a pre-existing dataset. Ensure every question’s reference answer exists verifiably within the currently specified knowledge base; otherwise, evaluation results will be inaccurate.
- Enter a name for the evaluation dataset.
- Select task types. When generating a dataset, choose 2–8 task types. The system provides four built-in types by default: Factual, Analytical, Comparative, and Tutorial. You may also define custom types by clicking Add Task Type, then entering the type name, description, and example.
- Select the model used to generate evaluation tasks. To ensure high-quality output, only
qwen-maxandqwen-plusare supported. - After selecting the model, the page displays estimated average and maximum token consumption.
- Click Generate Evaluation Dataset, confirm configuration details in the pop-up dialog, then click Continue Generation.
- Wait for generation to complete. Once finished, click Next.
- Estimated Average Consumption is a reference value; actual usage is billed per your account statement.
- Estimated Maximum Consumption is a hard cost cap set to prevent unexpectedly long outputs; actual token usage is typically far below this limit.
Configuring Evaluation Rules
- Set Sampling Count per Category: Specifies how many questions to sample from each selected task type. The system randomly selects this number of questions per type for final evaluation.
- Select the Evaluation Model: To ensure accuracy, only
qwen-maxandqwen-plusare supported. - After model selection, estimated average and maximum token consumption appear below. Higher sampling counts increase token usage.
- Before full evaluation, you may run a Trial Run to preview evaluation behavior.
A trial run executes a full evaluation on a single randomly selected question—consuming minimal tokens.
Executing Evaluation
- After confirming dataset and rule configurations, click Launch Evaluation Task.
- In the confirmation dialog, review evaluation settings and estimated costs, then click Start Evaluation. Wait for completion.
- After evaluation finishes, click Add Application in the task list to include additional applications for comparative analysis. Total applications per task cannot exceed eight.
Evaluation Report Analysis
Scoring Mechanism
The system uses an LLM to score each agent response on a 1–5 scale. Scoring compares the agent’s output against the reference answer in the evaluation dataset, assessing accuracy, completeness, and relevance. Scoring rules:
| Score | Meaning |
|---|---|
| 5 points | Correct answer, excellent quality. |
| 4 points | Correct answer, good quality. |
| Below 4 points | Incorrect answer—system automatically performs root-cause analysis. |
Result Analysis
- Overall Accuracy Rate: Measures overall application performance.
Formula:Accuracy Rate = (Number of responses scoring ≥ 4) / (Total number of responses) × 100%.
For multi-application evaluations, results are visualized in comparative charts. - BadCase Analysis: By default, the BadCase list shows the Top-5 lowest-scoring incorrect items, sorted ascending by score. Click View All Data to see the full list.
- Optimization Recommendations: Based on root-cause analysis, the system provides concrete suggestions—e.g., refining prompts, adjusting retrieval settings, or optimizing knowledge base chunking.
- RAG Agent Evaluation: Displays per-task-type scores (e.g., Factual, Analytical).
Root-Cause Analysis
For BadCases scoring below 4, the system automatically identifies which stage of the RAG pipeline failed. Below are root-cause categories, meanings, and corresponding optimization guidance:
- Model Misunderstanding: Correct knowledge was retrieved, but ambiguous prompt instructions or insufficient model reasoning capability led to an incorrect answer.
→ Optimize: Clarify output requirements in the prompt or switch to a stronger model. - Poor Re-ranking: Correct knowledge chunks were retrieved but ranked too low to be included in the final context passed to the LLM.
→ Optimize: Adjust re-ranking parameters or increase the number of chunks passed to the model. - Ineffective Retrieval: Retrieval strategy returned too many or too few relevant chunks—often due to suboptimal configuration.
→ Optimize: Tune retrieval method (e.g., vector similarity threshold, keyword weighting) based on data characteristics. - Incomplete Chunking: During knowledge base creation, semantic units were split across multiple chunks—typically due to overly fine-grained chunking.
→ Optimize: Increase chunk length or enable semantic chunking. - Knowledge Not Retrieved: No relevant chunks were retrieved—either due to empty retrieval results or missing content in the knowledge base.
→ Optimize: Add missing domain knowledge to the knowledge base.
Best Practices
A single evaluation reflects application performance only at a specific point in time. To sustain high agent quality, integrate Automated Evaluation into daily development and operations workflows.
Establish Continuous Evaluation
Trigger an evaluation after any of the following changes:
- After knowledge base updates: Adding, modifying, or deleting knowledge may impact retrieval and answer quality.
- After prompt adjustments: Prompt changes directly affect model output behavior.
- After model switching or upgrading: Different models vary significantly in understanding and generation capabilities.
- After retrieval/re-ranking configuration changes: These settings directly influence RAG recall quality.
- Regular regression testing (weekly/monthly): Even without active changes, periodic evaluation helps detect subtle quality drift.
Build an Optimization Feedback Loop
Follow this iterative cycle:
- Identify BadCases: Focus on responses scoring below 4 in the evaluation report.
- Analyze root causes: Use the system-provided root-cause category to quickly pinpoint failure origin per BadCase.
- Apply targeted fixes: Modify the corresponding configuration—prompt, retrieval settings, or knowledge base.
- Re-evaluate for validation: Retest using the same evaluation dataset and compare pre/post-optimization accuracy rates.