///////////////////////////////////////////////////////////////////////////////

    Copyright (c) 2019, 2020 Oracle and/or its affiliates.

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.

///////////////////////////////////////////////////////////////////////////////

= Helidon MP Metrics Guide
:description: Helidon metrics
:keywords: helidon, metrics, microprofile, guide

This guide describes how to create a sample MicroProfile (MP) project
that can be used to run some basic examples using both built-in and custom metrics with Helidon MP.

== What you need

[width=50%,role="flex, sm7"]
|===
|About 30 minutes
|<<about/03_prerequisites.adoc,Helidon Prerequisites>>
|https://github.com/helm/helm[Helm]
|===

=== Create a sample Helidon MP project

Use the Helidon MP Maven archetype to create a simple project that can be used for the examples in this guide.

[source,bash,subs="attributes+"]
.Run the Maven archetype:
----
mvn archetype:generate -DinteractiveMode=false \
    -DarchetypeGroupId=io.helidon.archetypes \
    -DarchetypeArtifactId=helidon-quickstart-mp \
    -DarchetypeVersion={helidon-version} \
    -DgroupId=io.helidon.examples \
    -DartifactId=helidon-quickstart-mp \
    -Dpackage=io.helidon.examples.quickstart.mp
----

=== Using the built-in metrics

Helidon provides three scopes of metrics: base, vendor, and application.  Helidon automatically provides built-in base and vendor metrics.
Applications can use these metrics without additional configuration or code changes.  Here are the metric endpoints:

1. `/metrics/base` - Base metrics data as specified by the MicroProfile Metrics specification.
2. `/metrics/vendor` - Helidon-specific metrics data.
3. `/metrics/application` - Application-specific metrics data.

NOTE: The `/metrics` endpoint will return data for all scopes.

The following example will demonstrate how to use the built-in metrics.  All examples are executed
from the root directory of your project (helidon-quickstart-mp).

[source,bash]
.Build the application, skipping unit tests, then run it:
----
mvn package -DskipTests=true
java -jar target/helidon-quickstart-mp.jar
----

NOTE: Metrics can be returned in either text format (the default), or JSON.  The text format uses Prometheus Text Format,
see https://prometheus.io/docs/instrumenting/exposition_formats/#text-format-details.

[source,bash]
.Verify the metrics endpoint in a new terminal window:
----
curl http://localhost:8080/metrics
----

[source,text]
.Text response:
----
# TYPE base:classloader_current_loaded_class_count counter
# HELP base:classloader_current_loaded_class_count Displays the number of classes that are currently loaded in the Java virtual machine.
base:classloader_current_loaded_class_count 7511
# TYPE base:classloader_total_loaded_class_count counter
# HELP base:classloader_total_loaded_class_count Displays the total number of classes that have been loaded since the Java virtual machine has started execution.
base:classloader_total_loaded_class_count 7512
...
----

You can get the same data in JSON format.

[source,bash]
.Verify the metrics endpoint with an HTTP accept header:
----
curl -H "Accept: application/json"  http://localhost:8080/metrics
----

[source,json]
.JSON response:
----
{
  "base": {
    "classloader.currentLoadedClass.count": 7534,
    "classloader.totalLoadedClass.count": 7538,
    "classloader.totalUnloadedClass.count": 1,
    "cpu.availableProcessors": 4,
    "cpu.systemLoadAverage": 2.83349609375,
    "gc.PS MarkSweep.count": 2,
    "gc.PS MarkSweep.time": 77,
    "gc.PS Scavenge.count": 5,
    "gc.PS Scavenge.time": 37,
    "jvm.uptime": 727588,
    "memory.committedHeap": 284164096,
    "memory.maxHeap": 3817865216,
    "memory.usedHeap": 53283088,
    "thread.count": 44,
    "thread.daemon.count": 35,
    "thread.max.count": 44
  },
  "vendor": {
    "grpc.requests.count": 0,
    "grpc.requests.meter": {
      "count": 0,
      "meanRate": 0.0,
      "oneMinRate": 0.0,
      "fiveMinRate": 0.0,
      "fifteenMinRate": 0.0
    },
    "requests.count": 6,
    "requests.meter": {
      "count": 6,
      "meanRate": 0.008275992296704147,
      "oneMinRate": 0.01576418632772332,
      "fiveMinRate": 0.006695060022357365,
      "fifteenMinRate": 0.0036382699664488415
    }
  }
}
----

You can get a single metric by specifying the name in the URL path.

[source,bash]
.Get the Helidon `grpc.requests.meter` metric:
----
curl -H "Accept: application/json"  http://localhost:8080/metrics/vendor/grpc.requests.meter
----

[source,json]
.JSON response:
----
{
  "grpc.requests.meter": {
    "count": 0,
    "meanRate": 0.0,
    "oneMinRate": 0.0,
    "fiveMinRate": 0.0,
    "fifteenMinRate": 0.0
  }
}
----

NOTE: You cannot get the individual fields of a metric. For example, you cannot target http://localhost:8080/metrics/vendor/grpc.requests.meter.count.

=== Metrics metadata

Each metric has associated metadata that describes:

1. name: The name of the metric.
2. units: The unit of the metric such as time (seconds, millisecond), size (bytes, megabytes), etc.
3. type: The type of metric: `Counter`, `Timer`, `Meter`, `Histogram`, or `Gauge`.

You can get the metadata for any scope, such as `/metrics/base`, as shown below:

[source,bash]
.Get the metrics metadata using HTTP OPTIONS method:
----
 curl -X OPTIONS -H "Accept: application/json"  http://localhost:8080/metrics/base
----

[source,json]
.JSON response (truncated):
----
{
  "classloader.currentLoadedClass.count": {
    "unit": "none",
    "type": "counter",
    "description": "Displays the number of classes that are currently loaded in the Java virtual machine.",
    "displayName": "Current Loaded Class Count"
  },
...
  "jvm.uptime": {
    "unit": "milliseconds",
    "type": "gauge",
    "description": "Displays the start time of the Java virtual machine in milliseconds. This attribute displays the approximate time when the Java virtual machine started.",
    "displayName": "JVM Uptime"
  },
...
  "memory.usedHeap": {
    "unit": "bytes",
    "type": "gauge",
    "description": "Displays the amount of used heap memory in bytes.",
    "displayName": "Used Heap Memory"
  }
}
----


=== Application-specific metrics data

You can create application-specific metrics and integrate them with Helidon using CDI.
To add a new metric, simply annotate the JAX-RS resource with one of the metric annotations. Metrics can
be injected at the class, method, and field-levels.  This document shows examples of all three.

Helidon will automatically create and register annotated application metrics and store them in the application `MetricRegistry`, which
also contains the metric metadata. The metrics will exist for the lifetime of the application.
Each metric annotation has mandatory and optional fields. The name field, for example, is optional.

==== Method level metrics

There are three metrics that you can use by annotating a method:

1. `@Counted` - Register a `Counter` metric
2. `@Timed` - Register a `Timer` metric
3. `@Metered` - Register a `Meter` metric

The following example will demonstrate how to use the `@Counted` annotation to track the number of times
the `/cards` endpoint is called.

[source,java]
.Create a new class `GreetingCards` with the following code:
----
package io.helidon.examples.quickstart.mp;

import java.util.Collections;
import javax.enterprise.context.RequestScoped;
import javax.json.Json;
import javax.json.JsonBuilderFactory;
import javax.json.JsonObject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import org.eclipse.microprofile.metrics.annotation.Counted;

@Path("/cards") //<1>
@RequestScoped // <2>
public class GreetingCards {

  private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Collections.emptyMap());

  @GET
  @Produces(MediaType.APPLICATION_JSON)
  @Counted(name = "any-card")  // <3>
  public JsonObject anyCard() throws InterruptedException {
    return createResponse("Here are some random cards ...");
  }

  private JsonObject createResponse(String msg) {
    return JSON.createObjectBuilder().add("message", msg).build();
  }
}
----
<1> This class is annotated with `Path` which sets the path for this resource
as `/cards`.
<2> The `@RequestScoped` annotation defines that this bean is
request scoped.  The request scope is active only for the duration of
one web service invocation and it is destroyed at the end of that
invocation.
<3> The annotation `@Counted` will register a `Counter` metric for this method, creating it if needed.
The counter is incremented each time the anyCards method is called.  The `name` attribute is optional.

NOTE: For Metrics 1.1, you must set `monotonic` field to `true` to force the count to increment when entering the method.
The default behavior is to decrement when exiting the method.  Here is an example:
`@Counted(name = "any-card", monotonic = true)`.


[source,java]
.Update the `GreetApplication` class as follows:
----
@Override
public Set<Class<?>> getClasses() {
    return Set.of(GreetResource.class, GreetingCards.class); // <1>
}
----
<1> Add the `GreetingCards` class to the set of classes managed by Helidon for this application.


[source,bash]
.Build and run the application, then invoke the application endpoints below:
----
curl http://localhost:8080/cards
curl http://localhost:8080/cards
curl -H "Accept: application/json"  http://localhost:8080/metrics/application
----

[source,json]
.JSON response:
----
{
  "io.helidon.examples.quickstart.mp.GreetingCards.any-card":2 // <1>
}
----
<1> The any-card count is two, since you invoked the endpoint twice.

NOTE: Notice the counter is fully qualified.  You can remove the package prefix by using the `absolute=true` field in the `@Counted` annotation.
You must use  `absolute=false` for class-level annotations.

==== Additional method-level metrics

The `@Timed` and `@Metered` annotations can also be used with a method.  For the following example. you can just annotate the same method with these metrics.
When using multiple annoations on a method, you *must* give the metrics different names as shown below.

[source,java]
.Update the `GreetingCards` class with the following code:
----
package io.helidon.examples.quickstart.mp;

import java.util.Collections;
import javax.enterprise.context.RequestScoped;
import javax.json.Json;
import javax.json.JsonBuilderFactory;
import javax.json.JsonObject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import org.eclipse.microprofile.metrics.MetricUnits;
import org.eclipse.microprofile.metrics.annotation.Counted;
import org.eclipse.microprofile.metrics.annotation.Metered;
import org.eclipse.microprofile.metrics.annotation.Timed;

@Path("/cards")
@RequestScoped
public class GreetingCards {

  private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Collections.emptyMap());

  @GET
  @Produces(MediaType.APPLICATION_JSON)
  @Counted(name = "cardCount", absolute = true) //<1>
  @Metered(name = "cardMeter", absolute = true, unit = MetricUnits.MILLISECONDS) //<2>
  @Timed(name = "cardTimer", absolute = true, unit = MetricUnits.MILLISECONDS) //<3>
  public JsonObject anyCard() throws InterruptedException {
    return createResponse("Here are some random cards ...");
  }

  private JsonObject createResponse(String msg) {
    return JSON.createObjectBuilder().add("message", msg).build();
  }
}

----
<1> Specify a custom name for the `Counter` metric and set `absolute=true` to remove the path prefix from the name.
<2> Add the `@Metered` annotation to get a `Meter` metric.
<3> Add the `@Timed` annotation to get a `Timer` metric.

[source,bash]
.Build and run the application, then invoke the application endpoints below:
----
curl http://localhost:8080/cards
curl http://localhost:8080/cards
curl -H "Accept: application/json"  http://localhost:8080/metrics/application
----


[source,json]
.JSON response:
----
{
  "cardCount": 2,
  "cardMeter": {  // <1>
    "count": 2,
    "meanRate": 0.3664337145491488,
    "oneMinRate": 0.4,
    "fiveMinRate": 0.4,
    "fifteenMinRate": 0.4
  },
  "cardTimer": { // <2>
    "count": 2,
    "meanRate": 0.36649792432150535,
    "oneMinRate": 0.4,
    "fiveMinRate": 0.4,
    "fifteenMinRate": 0.4,
    "min": 12944,
    "max": 2078856,
    "mean": 1045900.0,
    "stddev": 1032956.0,
    "p50": 2078856.0,
    "p75": 2078856.0,
    "p95": 2078856.0,
    "p98": 2078856.0,
    "p99": 2078856.0,
    "p999": 2078856.0
  }
}
----
<1> The `Meter` metric includes the count field (it is a superset of `Counter`).
<2> The `Timer` metric includes the `Meter` fields (it is a superset of `Meter`).


==== Reusing metrics

You can share a metric across multiple endpoints by specifying the reusable field in the metric annotation as
demonstrated below.

[source,java]
.Update the `GreetingCards` class with the following code:
----
package io.helidon.examples.quickstart.mp;

import java.util.Collections;
import javax.enterprise.context.RequestScoped;
import javax.json.Json;
import javax.json.JsonBuilderFactory;
import javax.json.JsonObject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import org.eclipse.microprofile.metrics.annotation.Counted;

@Path("/cards")
@RequestScoped
public class GreetingCards {

  private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Collections.emptyMap());

  @GET
  @Produces(MediaType.APPLICATION_JSON)
  @Counted(name = "anyCard",absolute = true)
  public JsonObject anyCard() throws InterruptedException {
    return createResponse("Here are some cards ...");
  }

  @GET
  @Path("/birthday")
  @Produces(MediaType.APPLICATION_JSON)
  @Counted(name = "specialEventCard", absolute = true, reusable = true)  // <1>
  public JsonObject birthdayCard() throws InterruptedException {
    return createResponse("Here are some birthday cards ...");
  }

  @GET
  @Path("/wedding")
  @Produces(MediaType.APPLICATION_JSON)
  @Counted(name = "specialEventCard", absolute = true, reusable = true)  // <2>
  public JsonObject weddingCard() throws InterruptedException {
    return createResponse("Here are some wedding cards ...");
  }

  private JsonObject createResponse(String msg) {
    return JSON.createObjectBuilder().add("message", msg).build();
  }
}
----
<1>  The `/birthday` endpoint uses a `Counter` metric, named `specialEventCard`.
<2>  The `/wedding` endpoint uses the same `Counter` metric, named `specialEventCard`.


[source,bash]
.Build and run the application, then invoke the following endpoints:
----
curl  http://localhost:8080/cards/wedding
curl  http://localhost:8080/cards/birthday
curl  http://localhost:8080/cards
curl -H "Accept: application/json"  http://localhost:8080/metrics/application
----

[source,json]
.JSON response from `/metrics/application`:
----
{
"anyCard": 1,
"specialEventCard": 2  // <1>
}
----
<1> Notice that `specialEventCard` count is two, since you accessed `/cards/wedding` and `/cards/birthday`.

==== Class-level metrics

You can collect metrics at the class-level to aggregate data from all methods in that class using the same metric.
The following example introduces a metric to count all card queries.  In the following example, the method-level metrics are not
needed to aggregate the counts, but they are left in the example to demonstrate the combined output of all three metrics.

[source,java]
.Update the `GreetingCards` class with the following code:
----
package io.helidon.examples.quickstart.mp;

import java.util.Collections;
import javax.enterprise.context.RequestScoped;
import javax.json.Json;
import javax.json.JsonBuilderFactory;
import javax.json.JsonObject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import org.eclipse.microprofile.metrics.annotation.Counted;

@Path("/cards")
@RequestScoped
@Counted(name = "totalCards") // <1>
public class GreetingCards {

  private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Collections.emptyMap());

  @GET
  @Produces(MediaType.APPLICATION_JSON)
  @Counted(absolute = true) // <2>
  public JsonObject anyCard() throws InterruptedException {
    return createResponse("Here are some random cards ...");
  }

  @Path("/birthday")
  @GET
  @Produces(MediaType.APPLICATION_JSON)
  @Counted(absolute = true) // <3>
  public JsonObject birthdayCard() throws InterruptedException {
    return createResponse("Here are some birthday cards ...");
  }

  private JsonObject createResponse(String msg) {
    return JSON.createObjectBuilder().add("message", msg).build();
  }
}
----
<1> This class is annotated with `@Counted`, which aggregates count data from all the method that have a `Count` annotation.
<2> Use `absolute=true` to remove path prefix for method-level annotations.
<3> Add a method with a `Counter` metric to get birthday cards.

[source,bash]
.Build and run the application, then invoke the following endpoints:
----
curl http://localhost:8080/cards
curl http://localhost:8080/cards/birthday
curl -H "Accept: application/json"  http://localhost:8080/metrics/application
----

[source,json]
.JSON response from `/metrics/application`:
----
{
  "anyCard": 1,
  "birthdayCard": 1,
  "io.helidon.examples.quickstart.mp.totalCards.GreetingCards": 2  // <1>
}
----
<1> The `totalCards` count is a total of all the method-level `Counter` metrics.  Class level metric names are always
fully qualified.


==== Field-level metrics

Field level metrics can be injected into managed objects, but they need to be updated by the application code.
This annotation can be used on fields of type `Meter`, `Timer`, `Counter`, and `Histogram`.

The following example shows how to use a field-level `Counter` metric to track cache hits.

[source,java]
.Update the `GreetingCards` class with the following code:
----
package io.helidon.examples.quickstart.mp;

import java.util.Collections;
import java.util.Random;
import javax.enterprise.context.RequestScoped;
import javax.inject.Inject;
import javax.json.Json;
import javax.json.JsonBuilderFactory;
import javax.json.JsonObject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import org.eclipse.microprofile.metrics.Counter;
import org.eclipse.microprofile.metrics.annotation.Counted;
import org.eclipse.microprofile.metrics.annotation.Metric;

@Path("/cards")
@RequestScoped
@Counted(name = "totalCards")
public class GreetingCards {

  private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Collections.emptyMap());

  @Inject
  @Metric(name = "cacheHits", absolute = true) // <1>
  private Counter cacheHits;

  @GET
  @Produces(MediaType.APPLICATION_JSON)
  @Counted(absolute = true)
  public JsonObject anyCard() throws InterruptedException {
    updateStats(); // <2>
    return createResponse("Here are some random cards ...");
  }

  @Path("/birthday")
  @GET
  @Produces(MediaType.APPLICATION_JSON)
  @Counted(absolute = true)
  public JsonObject birthdayCard() throws InterruptedException {
    updateStats();  // <3>
    return createResponse("Here are some birthday cards ...");
  }

  private JsonObject createResponse(String msg) {
    return JSON.createObjectBuilder().add("message", msg).build();
  }

  private void updateStats() {
    if (new Random().nextInt(3) == 1) {
      cacheHits.inc(); // <4>
    }
  }
}
----
<1> A `Counter` metric field, `cacheHits`, is automatically injected by Helidon.
<2> Call `updateStats()` to update the cache hits.
<3> Call `updateStats()` to update the cache hits.
<4> Randomly increment the `cacheHits` counter.

[source,bash]
.Build and run the application, then invoke the following endpoints:
----
curl http://localhost:8080/cards
curl http://localhost:8080/cards
curl http://localhost:8080/cards/birthday
curl http://localhost:8080/cards/birthday
curl http://localhost:8080/cards/birthday
curl -H "Accept: application/json"  http://localhost:8080/metrics/application
----

[source,json]
.JSON response from `/metrics/application`:
----
{
  "anyCard": 2,
  "birthdayCard": 3,
  "cacheHits": 2, // <1>
  "io.helidon.examples.quickstart.mp.totalCards.GreetingCards": 5
}
----
<1> The cache was hit two times out of five queries.

==== Gauge metric

The metrics you have tested so far are updated in response to an application REST request, i.e GET `/cards`.  These
metrics can be declared in a request scoped class and Helidon will store the metric in the `MetricRegistry`, so the value persists
across requests. When GET `/metrics/application` is invoked, Helidon will return the current value of the metric stored in the `MetricRegistry`.
The `Gauge` metric is different from all the other metrics. The application must provide a getter to return the gauge value in an
application scoped class. When GET `/metrics/application` is invoked, Helidon will call the `Gauge` getter, store that value
in the `MetricsRegistry`, and return it as part of the metrics response payload.  So, the `Gauge` metric value is updated real-time, in response to the
get metrics request.

The following example demonstrates how to use a `Gauge` to track application up-time.

[source,java]
.Create a new `GreetingCardsAppMetrics` class with the following code:
----
package io.helidon.examples.quickstart.mp;

import java.time.Duration;
import java.util.concurrent.atomic.AtomicLong;
import javax.enterprise.context.ApplicationScoped;
import javax.enterprise.context.Initialized;
import javax.enterprise.event.Observes;
import org.eclipse.microprofile.metrics.annotation.Gauge;

@ApplicationScoped // <1>
public class GreetingCardsAppMetrics {

  private AtomicLong startTime = new AtomicLong(0); // <2>

  public void onStartUp(@Observes @Initialized(ApplicationScoped.class) Object init) {
    startTime = new AtomicLong(System.currentTimeMillis()); // <3>
  }

  @Gauge(unit = "TimeSeconds")
  public long appUpTimeSeconds() {
    return Duration.ofMillis(System.currentTimeMillis() - startTime.get()).getSeconds();  // <4>
  }
}
----
<1> This managed object must be application scoped to properly register and use the `Gauge` metric.
<2> Declare an `AtomicLong` field to hold the start time of the application.
<3> Initialize the application start time.
<4> Return the application `appUpTimeSeconds` metric, which will be included in the application metrics.


[source,java]
.Update the `GreetingCards` class with the following code to simplify the metrics output:
----
package io.helidon.examples.quickstart.mp;

import java.util.Collections;
import javax.enterprise.context.RequestScoped;
import javax.json.Json;
import javax.json.JsonBuilderFactory;
import javax.json.JsonObject;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import org.eclipse.microprofile.metrics.annotation.Counted;

@Path("/cards")
@RequestScoped
public class GreetingCards {

  private static final JsonBuilderFactory JSON = Json.createBuilderFactory(Collections.emptyMap());

  @GET
  @Produces(MediaType.APPLICATION_JSON)
  @Counted(name = "cardCount", absolute = true)
  public JsonObject anyCard() throws InterruptedException {
    return createResponse("Here are some random cards ...");
  }

  private JsonObject createResponse(String msg) {
    return JSON.createObjectBuilder().add("message", msg).build();
  }
}
----

[source,bash]
.Build and run the application, then invoke the application metrics endpoint:
----
curl -H "Accept: application/json"  http://localhost:8080/metrics/application
----

[source,json]
.JSON response from `/metrics/application`:
----
{
  "cardCount": 0,
  "io.helidon.examples.quickstart.mp.GreetingCardsAppMetrics.appUpTimeSeconds": 6 // <1>
}
----
<1> The application has been running for 6 seconds.

=== Integration with Kubernetes and Prometheus

The following example shows how to integrate the Helidon MP application with Kubernetes.

[source,bash]
.Stop the application and build the docker image:
----
docker build -t helidon-metrics-mp .
----

[source,yaml]
.Create the Kubernetes YAML specification, named `metrics.yaml`, with the following content:
----
kind: Service
apiVersion: v1
metadata:
  name: helidon-metrics // <1>
  labels:
    app: helidon-metrics
  annotations:
    prometheus.io/scrape: 'true' // <2>
spec:
  type: NodePort
  selector:
    app: helidon-metrics
  ports:
    - port: 8080
      targetPort: 8080
      name: http
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: helidon-metrics
spec:
  replicas: 1 // <3>
  template:
    metadata:
      labels:
        app: helidon-metrics
        version: v1
    spec:
      containers:
        - name: helidon-metrics
          image: helidon-metrics-mp
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
----
<1> A service of type `NodePort` that serves the default routes on port `8080`.
<2> An annotation that will allow Prometheus to discover and scrape the application pod.
<3> A deployment with one replica of a pod.


[source,bash]
.Create and deploy the application into Kubernetes:
----
kubectl apply -f ./metrics.yaml
----

[source,bash]
.Get the service information:
----
kubectl get service/helidon-metrics
----

[source,bash]
----
NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
helidon-metrics   NodePort   10.99.159.2   <none>        8080:31143/TCP   8s // <1>
----
<1> A service of type `NodePort` that serves the default routes on port `31143`.

[source,bash]
.Verify the metrics endpoint using port `30116`, your port will likely be different:
----
curl http://localhost:31143/metrics
----

NOTE: Leave the application running in Kubernetes since it will be used for Prometheus integration.

==== Prometheus integration

The metrics service that you just deployed into Kubernetes is already annotated with `prometheus.io/scrape:`.  This will allow
Prometheus to discover the service and scrape the metrics.  In this exercise, you will install Prometheus
into Kubernetes, then verify that it discovered the Helidon metrics in your application.

[source,bash]
.Install Prometheus and wait until the pod is ready:
----
helm install stable/prometheus --name metrics
export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
kubectl get pod $POD_NAME
----

You will see output similar to the following.  Repeat the `kubectl get pod` command until you see `2/2` and `Running`. This may take up to one minute.

[source,bash]
----
metrics-prometheus-server-5fc5dc86cb-79lk4   2/2     Running   0          46s
----

[source,bash]
.Create a port-forward so you can access the server URL:
----
kubectl --namespace default port-forward $POD_NAME 7090:9090
----

Now open your browser and navigate to `http://localhost:7090/targets`.  Search for helidon on the page and you will see your
Helidon application as one of the Prometheus targets.

==== Final cleanup

You can now delete the Kubernetes resources that were just created during this example.

[source,bash]
.Delete the Prometheus Kubernetes resources:
----
helm delete --purge metrics
----

[source,bash]
.Delete the application Kubernetes resources:
----
kubectl delete -f ./metrics.yaml
----

=== Summary

This guide demonstrated how to use metrics in a Helidon MP application using various combinations of
metrics and scopes.

* Access metrics for all three scopes: base, vendor, and application
* Configure application metrics at the class, method, and field-level
* Integrate Helidon metrics with Kubernetes and Prometheus

Refer to the following references for additional information:

* MicroProfile Metrics specification at https://github.com/eclipse/microprofile-metrics/releases/tag/2.0
* MicroProfile Metrics Javadoc at https://javadoc.io/doc/org.eclipse.microprofile.metrics/microprofile-metrics-api/2.0.0
* Helidon Javadoc at https://helidon.io/docs/latest/apidocs/index.html?overview-summary.html

