Evaluation tasks are the core functionality of application evaluation, enabling systematic assessment of application output quality. They support evaluation for both agent-based and workflow-based applications, combining automated evaluators with human labeling for multi-dimensional analysis.
Evaluation tasks are the core functionality of application evaluation, enabling systematic assessment of application output quality. They support evaluation for both agent-based and workflow-based applications, combining automated evaluators with human labeling for multi-dimensional analysis.
After creating an evaluation task, click Details on the right side of the task row to access the task detail page.
On the task detail page, you can view data details and metric statistics:
Displays detailed results for each evaluation sample, supporting operations such as data annotation and label configuration.
Displays aggregated metrics including overall score and evaluation progress, with sections for:
Tokens consumed by large language models during evaluation tasks are billed normally. You can view token usage on the Evaluation Tasks page.
Can I modify an evaluation task after creation?
After creation, the task configuration—including associated application and evaluation dataset—cannot be modified. However, you may add human labels at any time. To use different configurations, create a new evaluation task.
What is the purpose of the “No Application Association” option?
The “No Application Association” option is intended for pure human annotation scenarios. When selected, the system will not invoke any application automatically for evaluation—you’ll rely entirely on manual labeling and assessment.
How do evaluators and labels work together?
Evaluators provide automated scoring and are best suited for evaluation dimensions with clear rules or definitive ground truth. Labels support human annotation and are ideal for subjective judgment-based dimensions. Within a single evaluation task, you can combine automated scoring via evaluators with manual labeling to achieve comprehensive coverage across all evaluation dimensions.
Creating an Evaluation Task
1
Configure Basic Information
Navigate to the Evaluation Tasks page, click Create Evaluation Task, and configure the following basic information:
| Field | Description |
|---|---|
| Task Name | A custom name for the task (up to 50 characters). |
| Task Description | A brief description of the task’s purpose or use case (up to 200 characters). |
| Select Evaluation Dataset | Choose an evaluation dataset and its version from the list of published datasets. |
| Select Application | Choose how to associate the task with an application: No Application Association (default, for pure human annotation), Workflow, or Agent. |
| Evaluators | Add one or more automated evaluators for scoring; parameter mapping is required. |
| Labels | Add labels for human annotation (optional). |
2
Configure Evaluators (Optional)
After selecting an evaluator, you must configure parameter mapping—mapping evaluator variables to actual data sources.Configuration Requirements:
- All variables must be mapped before saving the configuration.
- Ensure mappings are accurate to avoid incorrect scoring results.
3
Complete Creation
After confirming all configurations, click Finish Creation to create the evaluation task.
Once initiated, an evaluation task’s configuration cannot be modified.
Managing Tasks
After creating an evaluation task, click Details on the right side of the task row to access the task detail page.
On the task detail page, you can view data details and metric statistics:
Data Details
Displays detailed results for each evaluation sample, supporting operations such as data annotation and label configuration.
- Evaluator Scoring Results: Shows automated scores from selected evaluators for each sample.
- Label Configuration: Allows you to add labels to the evaluation task.
- Annotation Modes:
- Standard Mode: Fields are displayed horizontally in a flat layout.
- Quick Annotation Mode: Clicking enables inline editing of custom labels—categorical labels appear as dropdowns, and text-type labels appear as input fields; changes are saved immediately.
Metric Statistics
Displays aggregated metrics including overall score and evaluation progress, with sections for:
- Overall Score Dashboard
- Evaluation Progress
- Score Summary
- Score Breakdown