///////////////////////////////////////////////////////////////////////////////

    Copyright (c) 2019, 2020 Oracle and/or its affiliates.

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.

///////////////////////////////////////////////////////////////////////////////

= Tracing
:description: Helidon Tracing Support
= :keywords: helidon, tracing

== Tracing Support
Helidon includes support for tracing through the `https://opentracing.io/[OpenTracing]` APIs.
Tracing is integrated with WebServer, gRPC Server, and Security.

Support for specific tracers is abstracted. Your application can depend on
the abstraction layer and provide a specific tracer implementation as a Java
`ServiceLoader` service.


== Configuring Tracing with Helidon SE

Declare the following dependency in your project to use the tracer abstraction:

[source,xml]
.Tracer Abstraction
----
<dependency>
    <groupId>io.helidon.tracing</groupId>
    <artifactId>helidon-tracing</artifactId>
</dependency>
----

=== Configuring Tracing with WebServer

To configure tracer with WebServer:

[source,java]
.Configuring OpenTracing `Tracer`
----
ServerConfiguration.builder()
                   .tracer(TracerBuilder.create("my-application")                    // <1>
                                 .collectorUri(URI.create("http://10.0.0.18:9411"))  // <2>
                                 .build())
                   .build()
----
<1> The name of the application (service) to associate with the tracing events
<2> The endpoint for tracing events, specific to the tracer used, usually loaded from Config

=== Configuring Tracing with gRPC Server

[source,java]

.Configuring OpenTracing `Tracer`
----
Tracer tracer = (Tracer) TracerBuilder.create("Server")
        .collectorUri(URI.create("http://10.0.0.18:9411"))   // <1>
        .build();
----
<1> If using zipkin tracing system, the endpoint would be:
----
http://10.0.0.18:9411/api/v2/spans
----

.Configuring Tracing Attributes
----
GrpcTracingConfig tracingConfig = new GrpcTracingConfig.Builder()
                .withStreaming()
                .withVerbosity()
                .withTracedAttributes(ServerRequestAttribute.CALL_ATTRIBUTES,
                     ServerRequestAttribute.HEADERS,
                     ServerRequestAttribute.METHOD_NAME)
                .build();
----

.Configuring gRPC Server
----
GrpcServerConfiguration serverConfig = GrpcServerConfiguration.builder().port(0)
                .tracer(tracer)
                .tracingConfig(tracingConfig)
                .build();
----

=== Configuration using Helidon Config [[Tracing-config]]
There is a set of common configuration options that this section describes. In addition each tracer implementation
may have additional configuration options - please see the documentation of each of them.

Each implementation may provide defaults for these options.

All common configuration options:
|===
|key |description

|service |Name of the service sending the tracing information. This is usually visible in the trace data to
                distinguish actors in a conversation (e.g. when multiple microservices are connected together)
|protocol |Protocol of the tracing collector (e.g. `http`, `https`)
|host |Host of the tracing collector (e.g. `localhost`)
|port |Port of the tracing collector (e.g. `9411`)
|path |Path of the tracing collector service that is used to send spans to
|enabled |If set to false, tracing would be disabled
|tags |String tags that are to be added to each span reported (object node of string-string pairs)
|boolean-tags |Boolean tags that are to be added to each span reported (object node of string-boolean pairs)
|int-tags |Int tags that are to be added to each span reported (object node of string-int pairs)

|===

== Creating custom spans
To create a custom span that is a child of the WebServer request:

[source,java]
----
Span span = request.tracer()
        .buildSpan("my-operation")
        .asChildOf(request.spanContext())
                         .start();
----


== Trace propagation across services
Automated trace propagation is supported currently only with Jersey client.

[source,java]
.Tracing propagation with Jersey client
----
//...

Response response = client.target(serviceEndpoint)
    .request()
    .get();
----

== Traced spans
The following table lists all spans traced by Helidon components:

|===
|component      |span name              |description

|`web-server`   |`HTTP Request`         |The overall span of the Web Server from request intitiation until response
                                            Note that in `Zipkin` the name is replaced with `jax-rs` span name if `jax-rs` tracing
                                            is used.
|`web-server`   |`content-read`         |Span for reading the request entity
|`web-server`   |`content-write`        |Span for writing the response entity
|`security`     |`security`             |Processing of request security
|`security`     |`security:atn`         |Span for request authentication
|`security`     |`security:atz`         |Span for request authorization
|`security`     |`security:response`    |Processing of response security
|`security`     |`security:outbound`    |Processing of outbound security
|`jax-rs`       |A generated name       |Span for the resource method invocation, name is generated from class and method name
|`jax-rs`       |`jersey-client-call`   |Span for outbound client call
|===

Some of these spans `log` to the span. These log events can be (in most cases) configured:

|===
|span name          |log name               |configurable   |enabled by default |description

|`HTTP Request`     |`handler.class`        |YES            |YES                |Each handler has its class and event logged
|`security`         |`status`               |YES            |YES                |Logs either "status: PROCEED" or "status: DENY"
|`security:atn`     |`security.user`        |YES            |NO                 |The username of the user if logged in
|`security:atn`     |`security.service`     |YES            |NO                 |The name of the service if logged in
|`security:atn`     |`status`               |YES            |YES                |Logs the status of security response (such as `SUCCESS`)
|`security:atz`     |`status`               |YES            |YES                |Logs the status of security response (such as `SUCCESS`)
|`security:outbound`|`status`               |YES            |YES                |Logs the status of security response (such as `SUCCESS`)
|===

There are also tags that are set by Helidon components. These are not configurable.

|===
|span name          |tag name           |description

|`HTTP Request`         |`component`        |name of the component - `helidon-webserver`, or `jaxrs` when using MP
|`HTTP Request`         |`http.method`      |HTTP method of the request, such as `GET`, `POST`
|`HTTP Request`         |`http.status_code` |HTTP status code of the response
|`HTTP Request`         |`http.url`         |The path of the request (for SE without protocol, host and port)
|`HTTP Request`         |`error`            |If the request ends in error, this tag is set to `true`, usually accompanied by logs with details
|`content-read`         |`requested.type`   |Type (class) of the requested entity (if entity is read)
|`content-write`        |`response.type`    |Type (class) of the entity being sent (if enitty is sent)
|`security`             |`security.id`      |ID of the security context created for this request (if security is used)
|`jersey-client-call`   |`http.method`      |HTTP method of the client request
|`jersey-client-call`   |`http.status_code` |HTTP status code of client response
|`jersey-client-call`   |`http.url`         |Full URL of the request (such as `http://localhost:8080/greet`)
|===

=== Traced spans configuration

Each component and its spans can be configured using Config.
The traced configuration has the following layers:

- `TracingConfig` - the overall configuration of traced components of Helidon
- `ComponentTracingConfig` - a component of Helidon that traces spans (such as `web-server`, `security`, `jax-rs`)
- `SpanTracingConfig` - a single traced span within a component (such as `security:atn`)
- `SpanLogTracingConfig` - a single log event on a span (such as `security.user` in span `security:atn`)

The components using tracing configuration use the
`TracingConfigUtil`. This uses the `io.helidon.common.Context` to retrieve
current configuration.


==== Configuration using builder
Builder approach, example that disables a single span log event:

[source,java]
.Configure tracing using a builder
----
TracingConfig.builder()
     .addComponent(ComponentTracingConfig.builder("web-server")
             .addSpan(SpanTracingConfig.builder("HTTP Request")
                     .addSpanLog(SpanLogTracingConfig.builder("content-write").enabled(false).build())
                     .build())
             .build())
     .build()
----

==== Configuration using Helidon Config
Tracing configuration can be defined in a config file.

[source, yaml]
.Tracing configuration
----
tracing:
    components:
      web-server:
        spans:
          - name: "HTTP Request"
            logs:
              - name: "content-write"
                enabled: false
----

[source,java]
.Use the configuration in web server
----
routing.register(WebTracingConfig.create(config.get("tracing")));
----

==== Path based configuration in Helidon Web Server
For Web Server we have a path based support for configuring tracing, in addition
to the configuration described above.

Configuration of path can use any path string supported by the
Web Server. The configuration itself has the same possibilities
 as traced configuration described above. The path specific configuration
 will be merged with global configuration (path is the "newer" configuration, global is the "older")

[source,yaml]
.Configuration in YAML
----
tracing:
  paths:
    - path: "/favicon.ico"
      enabled: false
    - path: "/metrics"
      enabled: false
    - path: "/health"
      enabled: false
    - path: "/greet"
      components:
        web-server:
          spans:
          - name: "content-read"
            new-name: "read"
            enabled: false
----

[source,java]
.Configuration with Web Server
----
routingBuilder.register(WebTracingConfig.create(config.get("tracing"));
----

[source,java]
.Configuration with Web Server using a builder
----
routingBuilder.register(WebTracingConfig.builder()
    .addPathConfig(PathTracingConfig.builder()
        .path("/metrics")
        .tracingConfig(TracingConfig.DISABLED)
        .build();
    .build());
----

==== Renaming top level span using request properties
To have a nicer overview in search pane of a tracer, you can customize the top-level span name using configuration.

Example:
[source,yaml]
.Configuration in YAML
----
tracing:
  components:
    web-server:
      spans:
      - name: "HTTP Request"
        new-name: "HTTP %1$s %2$s"
----

This is supported ONLY for the span named "HTTP Request" on component "web-server".

Parameters provided:
1. Method - HTTP method
2. Path - path of the request (such as '/greet')
3. Query - query of the request (may be null)
