With Application Observability, you can view end-to-end processing flows (e.g., vector generation, vector retrieval, and large model invocations) for Alibaba Cloud Bailian applications within your business space—and obtain metrics such as latency and token counts.
The multi-node architecture inside Alibaba Cloud Bailian applications introduces several challenges for subsequent development, including:
Agent applications, workflow applications, and Managed Agents applications.
When using Application Observability for the first time, click Application Observability Configuration in the upper-right corner of the Application Observability page and complete the following steps:
Navigate to Application Observability, then click Select Application to Observe in the upper-right corner. If your previously created application does not appear in the list, it may be because:
In the application details page, click Export Data in the upper-right corner of the Trace List tab to export data matching your current filters—available in JSONL or Excel format.
In the application details page, click the Monitoring Statistics tab to view performance monitoring metrics for the application, including:
Application Observability supports directly adding Span data to an evaluation set for subsequent application evaluation. This enables you to use real production traffic as evaluation samples—building evaluation sets that better reflect actual business scenarios.
- Tracing internal application call chains
- Viewing model response latency
- Inspecting model reasoning processes
Application Observability currently does not provide an API.
Supported Applications
Agent applications, workflow applications, and Managed Agents applications.
Application Observability currently does not support agent applications created via the Assistant API.
Getting Started
Prerequisites
When using Application Observability for the first time, click Application Observability Configuration in the upper-right corner of the Application Observability page and complete the following steps:
- Authorize the OpenTelemetry service role permissions for Alibaba Cloud Tracing Analysis.
- Activate the Alibaba Cloud Tracing Analysis OpenTelemetry service.
- Initialize the LogStore for Alibaba Cloud Tracing Analysis OpenTelemetry storage.
Perform these operations using your Alibaba Cloud primary account. Activation typically takes effect within minutes—but may experience slight delays during peak periods. If you need to use a RAM user, the primary account must grant the required permissions to that RAM user.
How to Use
1. Select the Application to Observe
Navigate to Application Observability, then click Select Application to Observe in the upper-right corner. If your previously created application does not appear in the list, it may be because:
- The application has not yet been published.
- The application does not belong to the current business space.
2. Start Observing
-
After adding the application, it appears in the Application Observability list. All prompts submitted to this application—and their corresponding data and metrics—are automatically traced and synchronized to Application Observability (on a per-minute basis).
Clicking Disable Observability stops synchronization of trace data. Upon re-adding the application, only newly generated data will be synchronized.
- Click View Details under the Actions column to enter the application details page. By default, the Trace List tab is displayed, showing summary information—including prompt content, output, latency, invocation timestamp, and token count—for all prompts submitted to this application over the past period (up to 30 days). You can search by Request ID, Trace ID, or Span ID, and filter results by custom time range.
- Click a specific node’s Trace ID to view detailed information—including raw data and annotation records (if available). Child nodes (if any) can be expanded.
3. Export Data
In the application details page, click Export Data in the upper-right corner of the Trace List tab to export data matching your current filters—available in JSONL or Excel format.
4. View Monitoring Statistics
In the application details page, click the Monitoring Statistics tab to view performance monitoring metrics for the application, including:
- Invocation Count: Total number of application invocations and trend chart over time.
- Total Tokens: Trend chart for total tokens (including input and output tokens).
- Average Invocation Latency: Average latency across all application invocations.
- Average Time to First Token: Time elapsed before the first token is returned in streaming scenarios.
Add to Evaluation Set
Application Observability supports directly adding Span data to an evaluation set for subsequent application evaluation. This enables you to use real production traffic as evaluation samples—building evaluation sets that better reflect actual business scenarios.
- In the Application Observability list, click View Details next to the target application to open the Span list page. Then click Batch Operations, and select the Span entries you want to add to the evaluation set (multiple selections supported).
- Click Add to Evaluation Set, which opens the configuration page:
- Target Evaluation Set and Target Evaluation Set Name: Choose an existing evaluation set or create a new one.
- Import Mode: Choose between Append Data or Full Overwrite.
- Field Mapping: Map fields from Span data to corresponding fields in the evaluation set.
- After completing configuration, click Start Import.
Field mapping supports selecting from the full set of Span parameters. Each evaluation set supports up to 50 field mappings.