Manual evaluation of large language model (LLM) applications is an application-level assessment method. It involves manually constructing an evaluation dataset tailored to specific business scenarios, manually analyzing and scoring the application’s responses, and generating an evaluation report.
Manual evaluation of large language model (LLM) applications is an application-level assessment method. It involves manually constructing an evaluation dataset tailored to specific business scenarios, manually analyzing and scoring the application’s responses, and generating an evaluation report.
First, download the evaluation dataset template and populate it according to the template specifications.
Once evaluation completes, the Evaluation Status displays Completed. Click Results in the Actions column to view detailed evaluation results.
Step 1: Prepare the Evaluation Dataset
First, download the evaluation dataset template and populate it according to the template specifications.
- Prompt: The instruction provided to the LLM — for example, a question, a textual description, or even a parameterized text string.
- Completion: The expected response corresponding to the Prompt — e.g., an answer or descriptive text.
- SessionId: A session identifier; you may define this freely.
Step 2: Upload the Evaluation Dataset
1
Navigate to the Evaluation Dataset Page
Go to the Evaluation Dataset page under Application Evaluation: Evaluation Dataset.
2
Create an Evaluation Dataset
Click Create Evaluation Dataset, enter a custom name, and upload your prepared evaluation dataset file.
Supported file extensions:
.xls, .xlsx. Maximum file size: 20 MB. Up to 10 files may be uploaded per batch.3
Confirm and Publish
After clicking Confirm, wait until the Import Status changes to Import Successful, then click Publish in the Actions column to publish the dataset.
Draft datasets cannot be used for application evaluation — they must be published first.
Step 3: Create an Evaluation Task
1
Select an Application
Go to the Manual Evaluation page and click Create Evaluation Task. From the Application Batch Evaluation dropdown menu, select a published Agent Application, then click Next.
Only published Agent Applications are currently supported.
Application Batch Evaluation: This mode requires selecting an evaluation dataset and is intended for end-to-end performance validation before application deployment.
2
Select an Evaluation Dataset
Choose an uploaded and published evaluation dataset, then click Next.
3
Select Evaluation Dimensions
Select one or more evaluation dimensions, then click Next.If no custom evaluation dimension template has been configured, you may select a built-in template.
4
Confirm and Start Evaluation
Customize the Task Name, review the full task details, and click Cost Details to view the estimated cost.Once verified, click Start Evaluation.
Estimated Cost: Application batch evaluation performs inference using the selected evaluation dataset on publicly available model resources, which may incur token-based API charges or consume tokens from your token package.Evaluation Cost = Tokens Generated × Model Unit Price
5
Perform Labeling
When the evaluation status becomes Pending Labeling, click the Label button in the Actions column to evaluate the application’s generated outputs.
“Labeling” refers to comparing the application’s output against the reference answers in the evaluation dataset and assigning qualitative ratings (e.g., “Poor”, “Fair”, “Good”) or numeric scores (e.g., 1–5). This process helps identify strengths and weaknesses across different usage scenarios.Compare the evaluation dataset’s ground truth with the application’s output, provide an overall rating (“Poor”, “Fair”, or “Good”), then click Save & Move to Next.
6
Complete Evaluation
After labeling all entries in the evaluation dataset, click Complete Evaluation & Submit to finalize the application evaluation.