Skip to main content

Create and Manage Datasets

Create, publish, and manage evaluation datasets

Evaluation datasets store and manage all evaluation task data. They can be created from an empty dataset or by uploading a local file.

Creating an Evaluation Dataset

Go to the Evaluation Datasets page and click Create Evaluation Dataset, then complete the following steps:
1

Fill in the basic information

Enter the evaluation dataset name (required, up to 50 characters) and description (optional, up to 200 characters).
2

Select the creation method

  • Empty dataset (default): Configure the table schema only, without importing data. After creation, you can add data on the dataset details page, and column settings remain editable.
  • Upload a local file: Upload a .xls, .xlsx, or .csv file. The dataset table schema is built from the file's column fields, and the file data is imported at the same time. Maximum file size: 20 MB.
3

Configure the table schema

When creating an empty dataset, select a scenario first, then edit the table schema:
ScenarioDescriptionDefault table schema
AgentDefines the dataset based on the input and output parameters of the selected agent applicationinput, ideal output
WorkflowDefines the dataset based on the input and output parameters of the workflow applicationinput, ideal output
TrajectoryDefines the dataset based on agent execution trajectories, evaluating tool calls and the execution processinput, reference_output, trajectory
CustomFreely define the table schemaColumn 1, Column 2, Column 3
For the Agent or Workflow scenario, you can select a specific agent or workflow and its version. The system generates the table schema from its input and output parameters.Each column supports configuring the column name, data type, required flag, and description. Data types include String, Number, Boolean, Object, Array, and Trajectory. Column names can only contain letters, digits, or underscores, and must start with a letter or underscore.
For creating datasets with trajectory data and how to add trajectory data, see Add Trajectory Evaluation Data.
4

Confirm creation

Click Confirm to finish creating the evaluation dataset. You can continue adding data on the dataset details page.
When uploading a local file, the table schema is automatically built from the file's column fields after upload, and the file rows become the initial dataset data. You can also download the data template first, fill it in, and then upload it.

Publishing an Evaluation Dataset

  1. On the dataset details page, click Publish to open the publish version dialog.
  2. Fill in the version description (optional, up to 100 characters). Version IDs start from V1 and increment. An evaluation dataset keeps up to 10 versions; older versions are automatically overwritten.
  3. Click Confirm Publish. The publish status changes to Publishing. Once publishing completes, the dataset is ready for evaluation tasks.
Draft datasets cannot be used for application evaluation and must be published first.

Managing Evaluation Datasets

  1. Version management: Manage dataset versions—the draft version and the 10 most recent versions are displayed.
  2. Content search: Search within the dataset by selecting All Fields or a specific dataset header field, then entering the search content.
  3. Edit headers: Modify column names and descriptions, and add new columns. Once the dataset contains data, the data type and required flag of existing columns cannot be modified.
  4. Add data: Click Add Data to add a single record or import in batch. Batch import supports .xls, .xlsx, and .csv files, with append or full-overwrite modes, and configurable mapping between file columns and dataset columns.
  5. Export: Export the current dataset data.
  6. Delete: Delete the evaluation dataset after confirming in the dialog.
A dataset must be published once it is complete. Draft evaluation datasets cannot be used for application evaluation.