It consists of the three major pillars below.

1. Metrics - numeric values that are collected and aggregated over a period of time.
2. Tracing - the activities that occur when a request/transaction occurs in the system from the point of entry to exit.
3. Logging - text records of activities that occurred with relevant information along with the timestamp.

Observability tools and platforms enable developers to monitor, troubleshoot, and optimize their applications effectively.

Provide observability in Ballerina

Ballerina services and any client connectors are observable by default. HTTP/HTTPS and SQL client connectors use semantic tags to make tracing and metrics monitoring more informative.

This guide demonstrates how to enable Ballerina observability to observe Ballerina services using different tools and platforms. We will use the example given below.

Example: Observe a Ballerina service

Create a new Ballerina package using the bal new command.

$ bal new observability_demo

Created new package 'observability_demo' at observability_demo.

Replace the contents of the main.bal file, which is located inside the newly created observability_demo directory, with the following code:

import ballerina/http;
import ballerina/log;

type Product record {|
    int id;
    string name;
    float price;
|};

type OrderRequest record {|
    int productId;
    int quantity;
|};

type Order record {|
    int orderId;
    int productId;
    int quantity;
    float totalPrice;
|};

// Sample data
map<Product> products = {
    "1": {id: 1, name: "Laptop", price: 1200.00},
    "2": {id: 2, name: "Smartphone", price: 800.00},
    "3": {id: 3, name: "Headphones", price: 150.00}
};

map<Order> orders = {};
int orderCount = 0;

@display {
    label: "Shopping Service"
}
service /shop on new http:Listener(8090) {

    // List available products.
    resource function get products() returns Product[] {
        log:printInfo("Fetching product list");
        return products.toArray();
    }

    // Add a new product.
    resource function post product(Product product) returns http:Created|http:Conflict|error? {
        log:printInfo("Adding a new product");

        if products.hasKey(product.id.toString()) {
            log:printError("Product already exists with product ID", id =product.id);
            http:Conflict errorResponse = {
                body:  string `Product already exists with product ID: ${product.id}`
            };
            return errorResponse;
        }

        products[product.id.toString()] = product;
        log:printInfo("Product added successfully.", product = product);
        http:Created response = {
            body: string `Product added successfully with product ID: ${product.id}`
        };
        return response;   
    }

    // Place a new order.
    resource function post 'order(OrderRequest orderRequest) returns http:Accepted|http:NotFound|error? {
        log:printInfo("Received order request");

        if !products.hasKey(orderRequest.productId.toString()) {
            log:printError("Product not found with product ID", id = orderRequest.productId);
            http:NotFound errorResponse = {
                body:  string `Product not found with product ID: ${orderRequest.productId.toString()}`
            };
            return errorResponse;
        }
        Product product = products.get(orderRequest.productId.toString());
        Order newOrder = {orderId: orderCount, productId: orderRequest.productId, quantity: orderRequest.quantity, totalPrice: product.price * orderRequest.quantity};
        orders[orderCount.toString()] = newOrder;
        orderCount += 1;

        log:printInfo("Order placed successfully.", 'order = newOrder);
        http:Accepted response = {
            body:  newOrder
        };
        return response;
    }

    // Get order details by ID.
    resource function get 'order/[int orderId]() returns http:Ok|http:NotFound|error? {
        log:printInfo("Fetching order details");

        if !orders.hasKey(orderId.toString()) {
            log:printError("Order not found with order ID", id = orderId);
            http:NotFound errorResponse = {
                body: string `Order not found with order ID: ${orderId}`
            };
            return errorResponse;
        }

        Order 'order =  orders.get(orderId.toString());
        log:printInfo("Order details fetched successfully", 'order = 'order);
        http:Ok response = {
            body:  'order
        };
        return response;
    }
}

Enable observability for a Ballerina project

Observability can be enabled in a Ballerina project by adding the following section to the Ballerina.toml file.

[build-options]
observabilityIncluded=true

Note: The above configuration is included by default in the Ballerina.toml file generated when initiating a new package using the bal new command.

Alternatively, we can pass the --observability-included flag with the bal run command to start a Ballerina program with observability enabled.

Setting up Ballerina runtime configurations

To enable observability (both metrics and tracing) in the Ballerina runtime, use the following configurations:

[ballerina.observe]
enabled = true
provider = <PROVIDER>

Metrics and tracing can be enabled separately as well by using the following configurations. Add additional configurations specific to the tool or platform you are using.

[ballerina.observe]
metricsEnabled=true
metricsReporter=<METRICS_REPORTER>
tracingEnabled=true
tracingProvider=<TRACING_PROVIDER>

Common tags in metrics and traces

When observability is enabled, the Ballerina runtime adds a set of common tags (labels) to the metrics and spans it records for service resource functions, service remote functions, client remote calls, client resource calls, workers, and the main function. You can use these tags to filter, group, and correlate telemetry data in your observability tool or platform.

A tag is added only when it applies to the observed function. For example, src.resource.path is added only for resource functions, and error is added only when the function returns an error. Boolean tags are set to true when they apply and are left out otherwise. They are never set to false.

Source tags

These tags describe the function being observed.

Note: The service name is the label of the @display annotation attached to the service (for example, Shopping Service in the example above). If the annotation is not present, the service base path is used (for example, /shop). If the service has no base path, a generated name in the <org>_<module>_svc_<index> format is used.

Entry point tags

These tags describe the entry point of the execution in which the function was observed. The entry point is the first service resource function, service remote function, or function that started the observation, such as main. Nested observations carry the same entry point tags, so a call made deep in the call stack can be traced back to the entry point that triggered it.

Error tag

For example, a request to the GET /shop/products resource in the example above is recorded with the following tags:

src.module                   = myorg/observability_demo:0.1.0
src.position                 = main.bal:38:5
src.object.name              = Shopping Service
src.service.resource         = true
src.resource.accessor        = get
src.resource.path            = /products
entrypoint.function.module   = myorg/observability_demo:0.1.0
entrypoint.service.name      = Shopping Service
entrypoint.function.name     = /products
entrypoint.resource.accessor = get

Note: Connectors such as the HTTP and SQL clients and listeners add their own tags on top of these common tags (for example, http.method, http.url, and http.status_code_group).

Observability tools and platforms supported by Ballerina

This outlines how to enable and configure observability in Ballerina for various tools and platforms. It provides a step-by-step guide for setting up monitoring, tracing, and logging using widely used observability solutions.

Observability tools and platforms help monitor and analyze application performance, identify issues, and ensure reliability. The following are the main observability tools and platforms supported by Ballerina:

• OpenTelemetry: A vendor-neutral standard for exporting traces and metrics over the OpenTelemetry Protocol (OTLP) to any compatible backend.

• Prometheus: A monitoring system and time-series database for metrics collection and alerting.

• Jaeger: A distributed tracing platform for monitoring and debugging microservices.

• Zipkin: A distributed tracing system to collect and look up trace data.

• New Relic: A full-stack observability platform for application performance monitoring (APM) and telemetry.

• Datadog: A cloud-based observability service offering monitoring, metrics, traces, and logging.

• Elastic Stack: A collection of tools (Elasticsearch, Logstash, Kibana) for centralized logging and analytics.

The following contains a guide to set up and observe Ballerina programs in each of the observability tools or platforms mentioned above.

• Observe Ballerina programs with OpenTelemetry
• Observe Ballerina programs with Prometheus
• Observe Ballerina programs with Jaeger
• Observe Ballerina programs with Zipkin
• Observe Ballerina programs with New Relic
• Observe Ballerina programs with Datadog
• Observe Ballerina programs with Elastic Stack
• Observe Ballerina programs with OpenSearch
• Observe Ballerina programs with Moesif