Graders

A grader is a core component of application evaluation, used to automatically assess output quality. Bailian provides a rich set of preset grader templates and also supports custom LLM graders and Code graders, helping you build an evaluation system tailored to your business scenarios.

​

Create from a Preset Grader Template

Alibaba Cloud Bailian offers multiple preset grader templates covering a variety of evaluation scenarios. Templates are organized into the following categories:

• General Quality: Evaluates basic quality metrics of responses.
• Agent: Specifically evaluates the various capabilities of Agent applications.
• Text Matching: Uses exact rules for text matching.
• Text Similarity: Computes a text-similarity score.
• Format Validation: Validates output format compliance.

Configuration Steps

1. Click the Create Grader button in the top-right corner of the graders page, and choose to create a grader from a preset template.
2. Click Create from Preset on the desired template card to quickly create a grader, adjusting the details as needed.
3. In the trial-run validation area, click Run to confirm the evaluation logic works as expected, then click Create to save.
4. Click View Details to see the parameter information of the current grader template.

​

Create a Custom Grader

Click the Create Grader button in the top-right corner of the graders page, then click Create Custom Grader to enter the creation page.

​

Create an LLM Grader

An LLM grader uses a large model to score application output—suitable for evaluation scenarios that require semantic understanding. Configuration Steps

1. On the custom grader creation page, fill in the basic information:
   • Grader name: Up to 50 characters.
   • Description: Up to 200 characters, describing the grader's purpose.
2. Select LLM as the creation method.
3. Select a model: Click the model dropdown to choose the large model used for evaluation. Evaluation models are free for a limited time.
4. Prompt: Design evaluation rules in the prompt editor; click Select Preset Grader to import rules.
5. Configure scoring rules:
   • Score range: Determines the grader's scoring scale. The score range is added to the system prompt, so keep the score range consistent with the system prompt rules when editing.
   • Pass threshold: Determines the Pass/Fail status. Score ≥ threshold is Pass; score < threshold is Fail.
   Configuration tips:

   • Fine-grained evaluation: Use a larger score range (e.g., 0-100) to distinguish quality differences more precisely.
   • Quick classification: Use a smaller score range (e.g., 0-1 or 1-5) for quick pass/fail judgments.
   • Threshold setting: Typically set to the midpoint of the score range.
6. Trial-run validation: Fill in test values for each referenced parameter, click Run to confirm the evaluation logic works as expected, then click Create to save.

​

Create a Code Grader

A Code grader implements evaluation logic with Python scripts—suitable for evaluation scenarios that require precise rule-based judgment. Configuration Steps

1. On the custom grader creation page, fill in the basic information:
   • Grader name: Up to 50 characters.
   • Description: Up to 200 characters, describing the grader's purpose.
2. Select Code as the creation method.
3. Input parameters:
   • Default parameters: The system presets two parameters, query and response, which can be modified or deleted as needed.
   • Add parameters: Click Add Variable to define function input parameters.
4. Execution function: Write a Python 3.10 function in the code editor.
   Code requirements:

   • The function signature must include all parameters defined in the input parameters.
   • The function must return a scoring result (numeric type, within the score range).
   • Error-handling logic is recommended.
5. Score range: Determines the grader's scoring scale. The score range is added to the system prompt, so keep the score range consistent with the system prompt rules when editing.
6. Pass threshold: Determines the Pass/Fail status. Score ≥ threshold is Pass; score < threshold is Fail.
   Configuration tips:

   • Fine-grained evaluation: Use a larger score range (e.g., 0-100) to distinguish quality differences more precisely.
   • Quick classification: Use a smaller score range (e.g., 0-1 or 1-5) for quick pass/fail judgments.
   • Threshold setting: Typically set to the midpoint of the score range.
7. Trial-run validation: Enter test data (the format must include all input parameters), click Run to view the score or error message, then click Create to save the grader. Example test data:
   Copy

   {
     "query": "What is artificial intelligence?",
     "response": "Artificial intelligence (AI) is a branch of computer science dedicated to building systems that can perform tasks typically requiring human intelligence."
   }
   

​

Create a Grader from an Evaluation Task

Creating a grader from an evaluation task uses the annotation results of a historical evaluation task to automatically abstract and summarize a new LLM grader. This is suitable when you already have manual annotation data and want to solidify annotation experience into automated evaluation rules. Configuration Steps

1. On the custom grader creation page, fill in the basic information:
   • Grader name: Up to 50 characters.
   • Description: Up to 200 characters, describing the grader's purpose.
2. Select From Evaluation Task as the creation method.
3. Select a model: Click the model dropdown to choose the large model used for evaluation. Evaluation models are free for a limited time.
4. Select an evaluation task: Choose a historical evaluation task with completed annotations from the dropdown. The system learns evaluation rules based on that task's annotation results.
5. Configure field mapping: Select evaluation dataset fields or application output fields from the dropdown. Preset parameters:
   • query (String): The user's input question or query—map it to the question field in the dataset (e.g., the Prompt field).
   • response (String): The application output to evaluate or a dataset field (e.g., the Completion field).
   • label_score (Number): The manually annotated score—map it to the grader output score used in the evaluation task.
   • query maps to a data field in the evaluation dataset.
   • label_score maps to the grader in the evaluation task (shown as "grader-name_version Auto").
   • Only evaluation tasks that have completed evaluation can be used to create graders; incomplete tasks show "The grader status in the evaluation task is not complete."
6. Task description (optional): Provide complete background information about the evaluation task, including:
   • System purpose (e.g., "medical Q&A system").
   • Target user group.
   • Use case (e.g., "for medical professionals, providing fast and accurate clinical Q&A").
   • This description is used to automatically generate evaluation criteria and the final evaluation prompt.
7. Grader language: Choose the grader's output language—Chinese or English.
8. Configure scoring rules:
   • Score range: Determines the grader's scoring scale.
   • Pass threshold: Determines the Pass/Fail status. Score ≥ threshold is Pass; score < threshold is Fail.
9. After completing the configuration, click Create in the top-right corner to save the grader.

​

Manage Graders

On the custom grader list page, you can filter graders by type or name, and manage specific graders.

• Filter and search: Filter graders by LLM or Code type, or enter a grader name (fuzzy match supported, up to 50 characters) in the search box.
• Edit: The edit page shows a list of historical versions; select a version and click Overwrite Current Draft to switch and view different version configurations.
• Copy: Create a copy based on the current latest version, automatically carrying over all configurations.
• Delete: Deleted graders cannot be recovered—proceed with caution. A grader used by an evaluation task cannot be deleted. Before deleting, confirm no evaluation task depends on it.

​

Use a Grader in an Evaluation Task

After a grader is created, you must add it to an evaluation task and configure parameter mapping for it to take effect.

1. When creating an evaluation task, click Add Grader and select the grader and its version.
   Each evaluation task supports up to 10 graders.
   Choose the grader type as Custom Grader or Preset Grader, then map fields such as query, context, and reference_response to the corresponding dataset fields in the field-mapping area, and click Complete Creation.
2. After selecting a grader, complete parameter mapping to tell the system "where the grader's parameters get their data." Suppose you created an LLM grader whose prompt references 3 variables:
   • query: The user's question.
   • reference: The reference answer.
   • response: The answer to be evaluated.
   Mapping configuration:

   Grader Parameter	Maps To	Description
   query	Dataset field: question	Get the user's question from the dataset's question field
   reference	Dataset field: reference answer	Get the reference answer from the dataset's "reference answer" field
   response	Model output or dataset field	Get the content to evaluate from the application's actual output or a dataset field

   • All variables must be fully mapped before you can proceed to create the evaluation task.
   • Incorrect mapping will prevent the grader from working properly—verify field names carefully.

​

View Evaluation Results

After an evaluation task completes, view the grader results on the task details page. For details, see Evaluation Tasks. The evaluation task list includes columns for Evaluation Status, Task Name, Dataset Name, Application & Score (Pass Rate), Grader Details Auto, Label Details Manual, Token Consumption, Evaluation Time, and Actions. The Grader Details Auto column shows each automatic grader's run status (in progress or completed) and pass rate. The Actions column provides Details, Terminate, and Delete links.

​

Frequently Asked Questions