Evaluation datasets store and manage all evaluation task data. They can be created either automatically or by manual upload. Currently, two types of evaluation datasets are supported: Dialogue Analysis and Knowledge-based Question Answering.
Evaluation datasets store and manage all evaluation task data and can be created either automatically or by manual upload. Currently, two types of evaluation datasets are supported: Dialogue Analysis and Knowledge-based Question Answering.
- Dialogue Analysis: Designed for human evaluation. Single-turn or multi-turn dialogues are constructed to enable manual scoring and assessment of application outputs.
- Knowledge-based Question Answering: Designed for automated evaluation. Questions, reference answers, and keywords are provided; large language models then automatically score the accuracy of application outputs based on keyword matching and answer correctness.
Evaluation Dataset Types
Dialogue Analysis
- Supported file formats:
.xls,.xlsx. - Field descriptions:
| Field | Description |
|---|---|
Prompt | User input—typically a question or instruction. |
Completion | Reference answer corresponding to the Prompt, used to evaluate the model’s actual generated response. |
SessionId | Identifies a complete dialogue session. Multiple rows sharing the same SessionId are treated as a continuous multi-turn conversation. |
- Example:
| Prompt (user) | Completion (assistant) | SessionId | Description |
|---|---|---|---|
| Where is the Asian Games held? | Hangzhou. | 49aa5bc0-76de-471d-9f50-c8f7710bbc9d | Multi-turn dialogue — Turn 1 |
| In which year? | 2023 | 49aa5bc0-76de-471d-9f50-c8f7710bbc9d | Multi-turn dialogue — Turn 2 |
Knowledge-based Question Answering
- Supported file formats:
.xls,.xlsx. - Field descriptions:
| Field | Description |
|---|---|
Prompt | The user's input question. |
Completion | The standard reference answer to the question. |
Keywords | Keywords that must appear in the answer; used for keyword-matching during automated evaluation. |
Creating an Evaluation Dataset
1
Navigate to the Evaluation Datasets Page
Go to the Application Evaluation page in the Bailian console.
2
Create an Evaluation Dataset
Click Create Evaluation Dataset, enter a custom name, and upload your prepared evaluation dataset file(s).
Supported file extensions:
.xls, .xlsx. Maximum file size: 20 MB. Up to 10 files may be uploaded per batch.3
Confirm Upload
After clicking Confirm, the uploaded file(s) will appear on the Evaluation Datasets page.
4
Publish the Evaluation Dataset
Once the Import Status changes to Import Successful, click Publish in the Actions column to publish the dataset.
Draft evaluation datasets cannot be used for application evaluation. They must be published before use.