quarkus-workshop/docs/tracing.adoc

= Tracing with MicroProfile OpenTracing
:experimental:

This exercise shows how your Quarkus application can utilize https://github.com/eclipse/microprofile-opentracing/blob/master/spec/src/main/asciidoc/microprofile-opentracing.asciidoc[Eclipse MicroProfile OpenTracing] to provide distributed tracing for interactive web applications.

In a distributed cloud-native application, multiple microservices are collaborating to deliver the expected functionality. If you have hundreds of services, how do you debug an individual request as it travels through a distributed system? For Java enterprise developers, the Eclipse MicroProfile OpenTracing specification makes it easier. Let's find out how.

== Install Jaeger

https://www.jaegertracing.io/[Jaeger], inspired by Dapper and OpenZipkin, is a distributed tracing system released as open source by Uber Technologies. It is used for monitoring and troubleshooting microservices-based distributed systems, including:

* Distributed context propagation
* Distributed transaction monitoring
* Root cause analysis
* Service dependency analysis
* Performance / latency optimization

Jaeger exposes a _collector_ through which apps (like our People app) report tracing details. Jaeger stores and reports on this tracing activity.

[NOTE]
====
**What are Traces and Spans?**

At the highest level, a _trace_ tells the story of a transaction or workflow as it propagates through a (potentially distributed) system. A trace is a directed acyclic graph (DAG) of _spans_: named, timed operations representing a contiguous segment of work in that trace.

Each component (microservice) in a distributed trace will contribute its own span or spans.
====

To install Jaeger, run the following command in your Terminal:

[source,sh,role="copypaste"]
----
oc process -f https://raw.githubusercontent.com/jaegertracing/jaeger-openshift/master/all-in-one/jaeger-all-in-one-template.yml | oc create -f -
----

Jaeger exposes its collector at different ports for different protocols. Most use the HTTP collector at `jaeger-collector:14268` but other protocols like gRPC are also supported on different ports.

== Add Tracing to Quarkus

Like other exercises, we'll need another extension to enable tracing. Install it with:

[source,sh,role="copypaste"]
----
mvn quarkus:add-extension -Dextensions="opentracing, rest-client"
----

This will add the necessary entries in your `pom.xml` to bring in the OpenTracing capability, and anm HTTP REST Client we'll use pater. 

== Configure Quarkus

Next, open the `application.properties` file (in the `src/main/resources` directory). Add the following lines to it to configure the default Jaeger tracer in Quarkus:

[source,none,role="copypaste"]
----
quarkus.jaeger.service-name=people # <1>
quarkus.jaeger.sampler-type=const  # <2>
quarkus.jaeger.sampler-param=1     # <2>
quarkus.jaeger.endpoint=http://jaeger-collector:14268/api/traces # <3>
----
<1> The name of our service from the perspective of Jaeger (useful when multiple apps report to the same Jaeger instance)
<2> How Jaeger samples traces. https://www.jaegertracing.io/docs/1.7/sampling/#client-sampling-configuration[Other options exist] to tune the performance.
<3> This is the default HTTP-based collector exposed by Jaeger

== Test it out

Like many other Quarkus frameworks, sensible defaults and out of the box functionality means you can get immediate value out of Quarkus without changing any code. By default, all JAX-RS endpoints (like our `/hello` and others) are automatically traced. Let's see that in action by re-deploying our traced app.

First, re-build the app using the command palette and selecting **Create Executable JAR**. Once that's done, run the following command to re-deploy:

[source,sh,role="copypaste"]
----
oc start-build people --from-file target/*-runner.jar --follow
----

== Confirm deployment

Run and wait for the app to complete its rollout:

[source,sh,role="copypaste"]
----
oc rollout status -w dc/people
----

== Trigger traces

You'll need to trigger some HTTP endpoints to generate traces, so let's re-visit our earlier DataTables-powered app. Run this command to open the table with our people database:

[source,sh,role="copypaste"]
----
echo http://$(oc get route people -o=go-template --template={% raw %}'{{ .spec.host }}'{% endraw %})/datatable.html
----

Open that URL in a separate browser tab, and exercise the table a bit by paging through the entries to force several RESTful calls back to our app:

image::paging.png[paging,600]

== Inspect traces

Use this command to generate the URL to Jaeger:

[source,sh,role="copypaste"]
----
echo http://$(oc get route jaeger-query -o=go-template --template={% raw %}'{{ .spec.host }}'{% endraw %})
----

Open that URL in a separate tab to arrive at the Jaeger Tracing console (leave the tab open as we'll use it later)

image::jaegerui.png[jaeger,600]

Using the menu on the left, select the `people` Service, and click **Find Traces**. Jaeger will show the collected traces on the right:

image::find1.png[jaeger,600]

Click on one of the traces from "a few seconds ago" to show the individual _spans_ of each trace:

image::trace1.png[jaeger,600]

You can see that this trace (along with the others) shows the incoming HTTP GET operation to the `/datatable` endpoint we created earlier, along with the time it took, and other ancillary info about the request. Not terribly interesting as it's a single call, but you can imagine with a real world app and multiple microservices working together, that traces could reveal a lot of detail.

[NOTE]
====
Service Mesh technologies like https://istio.io[Istio] can provide even more tracing prowess as the calls across different services are traced at the network level, not requiring _any_ frameworks or developer instrumentation to be enabled for tracing.
====

== Tracing external calls

This exercise showa how to use the https://github.com/eclipse/microprofile-rest-client[MicroProfile REST Client] with Quarkus in order to trace _external_, outbound requests with very little effort.

We will use the publicly available https://swapi.co[Star Wars API] to fetch some characters from the Star Wars universe. Our first order of business is to setup the model we will be using, in the form of a StarWarsPerson POJO.

=== Create model 

Create a new class in the `org.acme.people.model` package called `StarWarsPerson` with the following content:

[source,java,role="copypaste"]
----
package org.acme.people.model;

public class StarWarsPerson {

    private String name;
    private String mass;

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getMass() {
        return mass;
    }

    public void setMass(String mass) {
        this.mass = mass;
    }
}
----

This contains a subset of the full Star Wars model, just enough to demonstrate tracing.

=== Create interface 

Using the https://github.com/eclipse/microprofile-rest-client[MicroProfile REST Client] is as simple as creating an interface using the proper JAX-RS and MicroProfile annotations. Create a new Java class in the `org.acme.people.service` package called `StarWarsService` with the following content:

[source,java,role="copypaste"]
----
package org.acme.people.service;

import org.acme.people.model.StarWarsPerson;
import org.eclipse.microprofile.rest.client.annotation.ClientHeaderParam;
import org.eclipse.microprofile.rest.client.inject.RegisterRestClient;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;

@RegisterRestClient // <1>
@Path("/api") // <2>
public interface StarWarsService {

    @GET
    @Path("/people/{id}/") // <2>
    @Produces("application/json") // <3>
    @ClientHeaderParam(name="User-Agent", value="QuarkusLab") // <4>
    StarWarsPerson getPerson(@PathParam("id") int id); // <5>
}
----
<1> `@RegisterRestClient` allows Quarkus to know that this interface is meant to be available for CDI injection as a REST Client
<2> `@Path`, `@GET` and `@PathParam` are the standard JAX-RS annotations used to define how to access the service
<3> While `@Consumes` and `@Produces` are optional as auto-negotiation is supported, it is heavily recommended to annotate your endpoints with them to define precisely the expected content types. It will also allow to narrow down the number of JAX-RS providers (which can be seen as converters) included in the native executable.
<4> The Star Wars API requires a `User-Agent` header, so with Quarkus we add that with `@ClientHeaderParam`. Other parameters can be added here as needed.
<5> The `getPerson` method gives our code the ability to query the Star Wars API by `id`. The client will handle all the networking and marshalling leaving our code clean of such technical details.

=== Configure endpoint

In order to determine the base URL to which REST calls will be made, the REST Client uses configuration from `application.properties`. To configure it, add this to your `application.properties` (in `src/main/resource`):

[source,none,role="copypaste"]
----
org.acme.people.service.StarWarsService/mp-rest/url=https://swapi.co
----

Having this configuration means that all requests performed using our code will use `https://swapi.co` as the base URL.

Note that `org.acme.people.service.StarWarsService` must match the fully qualified name of the StarWarsService interface we created in the previous section.

Using the configuration above, calling the `getPerson(int)` method of StarWarsService with a value of `1` would result in an HTTP GET request being made to `https://swapi.co/api/people/1/` (you can use `curl https://swapi.co/api/people/1/ | jq` to verify this URL produces a real character)

=== Final step: add endpoint

We need to `@Inject` an instance of our new `StarWarsService` and call it. Open the existing `PersonResource` class and add the following injected field and method:

[source,java,role="copypaste"]
----
@Inject
@RestClient
StarWarsService swService; // <1>

@GET
@Path("/swpeople")
@Produces(MediaType.APPLICATION_JSON)
public List<StarWarsPerson> getCharacters() {
    return Arrays.stream(new int[] {1, 2, 3, 4, 5}) // <2>
        .mapToObj(swService::getPerson)  // <3>
        .collect(Collectors.toList());  // <4>
}
----
<1> Our injected service
<2> Generate a stream of 5 integers that we will use as IDs to pass to the service
<3> For each of the integers, call the `StarWarsService::getPerson` method
<4> Collect the results into a list and return it

== Test it out

First, re-build the app using the command palette and selecting **Create Executable JAR**. Once that's done, run the following command to re-deploy:

[source,sh,role="copypaste"]
----
oc start-build people --from-file target/*-runner.jar --follow
----

== Confirm deployment

Run and wait for the app to complete its rollout:

[source,sh,role="copypaste"]
----
oc rollout status -w dc/people
----

== Trigger traces

Try out our new endpoint by running the following command:

[source,sh,role="copypaste"]
----
curl http://$(oc get route people -o=go-template --template={% raw %}'{{ .spec.host }}'{% endraw %})/person/swpeople
----

You should see:

[source,json]
----
[
  {
    "mass": "77",
    "name": "Luke Skywalker"
  },
  {
    "mass": "75",
    "name": "C-3PO"
  },
  {
    "mass": "32",
    "name": "R2-D2"
  },
  {
    "mass": "136",
    "name": "Darth Vader"
  },
  {
    "mass": "49",
    "name": "Leia Organa"
  }
]
----

== Inspect traces

Go back to the Jaeger console tab, and click **Find Traces**. The new trace should appear the top with multiple spans. Click on it to display details:

image::swpeople.png[swpeople,800]

You can see that this trace (along with the others) shows multiple spans: the incoming HTTP GET operation to the `/swperson` endpoint we created earlier, and the external calls to the Star Wars API. Expand the traces to show the detail:

image::swpeopleext.png[swpeopleext,800]

== Extra credit: Explicit method tracing

An annotation is provided to define explicit Span creation. This works on top of the "no-action" setup we did in the previous steps.

The `@Traced` annotation, applies to a class or a method. When applied to a class, the `@Traced` annotation is applied to all methods of the class. If the annotation is applied to a class and method then the annotation applied to the method takes precedence. The annotation starts a Span at the beginning of the method, and finishes the Span at the end of the method.

If you have time after this workshop, add a `@Traced` annotation to some of the other methods and test them out.

== Congratulations!

You've seen how to enable automatic tracing for JAX-RS methods as well as create custom tracers for non-JAX-RS methods and external services by using MicroProfile OpenTracing. This specification makes it easy for Quarkus developers to instrument services with distributed tracing for learning, debugging, performance tuning, and general analysis of behavior.