Skip to main content

Observability

View end-to-end application processing flows and obtain monitoring metrics such as latency, tokens, and QPM.

The multi-node architecture inside Alibaba Cloud Bailian applications introduces several challenges for subsequent development, including:
  • Tracing internal application call chains
  • Viewing model response latency
  • Inspecting model reasoning processes
Using the Application Observability feature, you can observe end-to-end processing flows (e.g., large model invocations, MCP invocations, and vector retrieval) for Alibaba Cloud Bailian applications in your business space—and view monitoring metrics such as latency and token counts, as well as rate-limiting metrics such as QPM (updated on a per-minute basis).
Application Observability currently does not provide an API.

Supported Applications

Agent applications (including Agent 1.0 and Agent 2.0) and workflow applications.
Application Observability currently does not support agent applications created via the Assistant API.

Getting Started

Prerequisites

When using Application Observability for the first time, click Application Observability Configuration in the upper-right corner of the Application Observability page and complete the following steps:
  1. Authorize the OpenTelemetry service role permissions for Alibaba Cloud Tracing Analysis.
  2. Activate the Alibaba Cloud Tracing Analysis OpenTelemetry service.
  3. Initialize the LogStore for Alibaba Cloud Tracing Analysis OpenTelemetry storage.
Perform these operations using your Alibaba Cloud primary account. Activation typically takes effect within minutes—but may experience slight delays during peak periods. If you need to use a RAM user, the primary account must grant the required permissions to that RAM user.

How to Use

1. Enable Application Observability

Navigate to Application Observability, find the target application in the list, and click Enable Observability. If your previously created application does not appear in the list, it may be because:
  • The application has not yet been published.
  • The application does not belong to the current business space.

2. Start Observing

  1. Once observability is enabled for an application, all prompts submitted to this application—and their corresponding data and metrics—are automatically traced and synchronized to Application Observability (on a per-minute basis).
  2. Click View Details under the Actions column to enter the application details page. By default, the Trace List tab is displayed, showing summary information—including prompt content, output, latency, invocation timestamp, and token count—for all prompts submitted to this application over the past period (up to 30 days). You can search by Request ID, Trace ID, or Span ID, and filter results by custom time range.
  3. Click a specific node’s Trace ID to view detailed information—including raw data and annotation records (if available). Child nodes (if any) can be expanded.
Clicking Disable Observability stops synchronization of trace data. Upon re-enabling, only newly generated data will be synchronized.

3. Export Data

In the application details page, click Export Data in the upper-right corner of the Trace List tab to export data matching your current filters—available in JSONL or Excel format.

4. View Monitoring Statistics

In the application details page, click the Monitoring Statistics tab to view and download performance monitoring data for the application, including:
  • Overview: Invocation count, total tokens, average invocation latency, average time to first token, and error count—each with period-over-period comparison.
  • Invocation Analysis: Trend charts for invocation count, error count, and average invocation latency.
  • Token Analysis: Total tokens (all / input / output), average input tokens per request, average output tokens per request, average time to first token, average invocation latency, and per-model token consumption trends.
  • Model Analysis: Per-model invocation distribution, invocation trends, error trends, and TTFT trends.
  • Tool Invocation Analysis: Per-tool invocation distribution, invocation trends, latency trends, and error count trends.
You can customize the time range (up to 30 days) and aggregation granularity (per minute, per hour, or per day).

5. View Rate-Limiting Statistics

In the application details page, click the Rate-Limiting Statistics tab to view and download rate-limiting monitoring data for the application, including:
  • Application: QPM throttling, QPM throttling counts, and more.
  • Quick configuration of alert rules, and support for filing a ticket to request a rate-limit increase.
For details on configuring rate-limiting alert rules, see Alert Management.

Add to Evaluation Set

Application Observability supports directly adding Trace data to an evaluation set for subsequent application evaluation. This enables you to use real production traffic as evaluation samples—building evaluation sets that better reflect actual business scenarios.
  1. In the Application Observability list, click View Details next to the target application to open the Trace list page. Then click Batch Operations, and select the Trace entries you want to add to the evaluation set (multiple selections supported).
  2. Click Add to Evaluation Set and configure the following:
    • Target evaluation set: Choose an existing evaluation set, or create a new one (enter the dataset name).
    • Import mode: Append Data (adds new data to the end of the target dataset) or Full Overwrite (clears the target dataset before writing).
    • Field mapping: Map Trace fields to evaluation set fields. The Trace input and output can be mapped directly to the corresponding columns. When Trajectory is selected, the Trace data is cleaned into the trajectory structure before writing, and the target dataset column must be of the Trajectory type.
  3. After completing configuration, click Start Import. The import result is displayed when the import completes; for failed records, download the log to check the reasons.
The data type of the Trace field and the evaluation set field must match; records with mismatched types cannot be imported.
After backfilling Trace trajectory data as evaluation samples, you can run trajectory evaluation with trajectory graders. See Add Trajectory Evaluation Data.
Field mapping supports selecting from the full set of Span parameters. Each evaluation set supports up to 50 field mappings.