= Monitoring Quarkus Apps using MicroProfile Metrics
:experimental:

This exercise demonstrates how your Quarkus application can utilize the https://github.com/eclipse/microprofile-metrics[MicroProfile Metrics] specification through the SmallRye Metrics extension.

MicroProfile Metrics allows applications to gather various metrics and statistics that provide insights into what is happening inside the application. They ey serve to pinpoint issues, provide long term trend data for capacity planning and pro-active discovery of issues (e.g. disk usage growing without bounds). Metrics can also help those scheduling systems decide when to scale the application to run on more or fewer machines.

The metrics can be read remotely using JSON format or the https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format[OpenMetrics text format], so that they can be processed by additional tools such as Prometheus, and stored for analysis and visualisation. You can then use tools like http://prometheus.io[Prometheus] and http://grafana.com[Grafana] to collect and display metrics for your Quarkus apps.

== Install Prometheus

First, let's install Prometheus. Prometheus is an open-source systems monitoring and alerting toolkit featuring:

* a multi-dimensional https://prometheus.io/docs/concepts/data_model/[data model] with time series data identified by metric name and key/value pairs
* https://prometheus.io/docs/prometheus/latest/querying/basics/[PromQL], a flexible query language to leverage this dimensionality
* time series collection happens via a pull model over HTTP

To install it, first create a Kubernetes ConfigMap that will hold the Prometheus configuration. In the Terminal, run the following:

[source,sh,role="copypaste"]
----
oc create configmap prom --from-file=prometheus.yml=src/main/kubernetes/prometheus.yml
----

This will create a ConfigMap using the contents of the `src/main/kubernetes/prometheus.yml` file in your project (we've created this file for you). It contains basic Prometheus configuration, plus a specific _target_ which instructs it to look for application metrics from both Prometheus itself, and our `people` app, on HTTP port 8080 at the `/metrics` endpoint. Here's a snippet of that file:

[source,yml]
----
scrape_configs:
  - job_name: 'prometheus' # <1>
    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'people_app' # <2>
    static_configs:
    - targets: ['people:8080'] 
----
<1> Configures Prometheus to scrape metrics from itself
<2> COnfigures Prometheus to scrape metrics from Kubernetes service `people` on port `8080` with HTTP, at the default metrics endpoint of `/metrics`

Next, deploy and expose Prometheus using its public Docker Hub image:

[source,sh,role="copypaste"]
----
oc new-app prom/prometheus && oc expose svc/prometheus
----

And finally, mount the ConfigMap into the running container:

[source,sh,role="copypaste"]
----
oc set volume dc/prometheus --add -t configmap --configmap-name=prom -m /etc/prometheus/prometheus.yml --sub-path=prometheus.yml
----

This will cause the contents of the ConfigMap's `prometheus.yml` data to be mounted at `/etc/prometheus/prometheus.yml` where Prometheus is expecting it.

Verify Prometheus is up and running:

[source,sh,role="copypaste"]
----
oc rollout status -w dc/prometheus
----

You should see `replication controller "prometheus-2" successfully rolled out`.

== Add Metrics to Quarkus

Like other exercises, we'll need another extension to enable metrics. Install it with:

[source,sh,role="copypaste"]
----
mvn quarkus:add-extension -Dextensions="metrics"
----

This will add the necessary entries in your `pom.xml` to bring in the Metrics capability. It will import the `smallrye-metrics` extension which is an implementation of the MicroProfile Metrics specification used in Quarkus.

== Start the app

Next, Run the app once more by using the command palette and select **Build and Run Locally** to start the app up in dev local mode.

== Test Metrics endpoint

You will be able to immediately see the raw metrics generated from Quarkus apps. Run this in the Terminal:

[source,sh,role="copypaste"]
----
curl http://localhost:8080/metrics
----

You will see a bunch of metrics in the OpenMetrics format:

[source, none]
----
# HELP base:jvm_uptime_seconds Displays the time from the start of the Java virtual machine in milliseconds.
# TYPE base:jvm_uptime_seconds gauge
base:jvm_uptime_seconds 5.631
# HELP base:gc_ps_mark_sweep_count Displays the total number of collections that have occurred. This attribute lists -1 if the collection count is undefined for this collector.
# TYPE base:gc_ps_mark_sweep_count counter
base:gc_ps_mark_sweep_count 2.0
----

This is what Prometheus will use to access and index the metrics from our app when we deploy it to the cluster.

== Add additional metrics

Out of the box, you get a lot of basic JVM metrics which are useful, but what if you wanted to provide metrics for your app? Let's add a few using the MicroProfile Metrics APIs.

Open the `GreetingService` class (in the `org.acme.people.service` package). Let's add a metric to count the number of times we've greeted someone. Add the following annotation to the `greeting()` method:

[source,java,role="copypaste"]
----
@Counted(name = "greetings", monotonic = true, description = "How many greetings we've given.")
----

(You'll need to import the new `Counted` class using _Assistant > Organize Imports_).

Trigger a greeting:

[source,sh,role="copypaste"]
----
curl http://localhost:8080/hello/greeting/quarkus
----

And then access the metrics again, this time we'll look for our new metric, specifying a scope of _application_ in the URL so that only metrics in that _scope_ are returned:

[source,sh,role="copypaste"]
----
curl http://localhost:8080/metrics/application
----

You'll see:

[source, none]
----
# HELP application:org_acme_people_service_greeting_service_greetings How many greetings we've given.
# TYPE application:org_acme_people_service_greeting_service_greetings counter
application:org_acme_people_service_greeting_service_greetings 1.0
----

This shows we've accessed the greetings once (`1.0`). Repeat the `curl` greeting a few times and then access metrics again, and you'll see the number rise.

[NOTE]
====
The comments in the metrics output starting with `#` are part of the format and give human-readable descriptions to the metrics which you'll see later on.
====

[NOTE]
====
In the OpenMicroProfile Metrics names are prefixed with things like `vendor:` or `application:` or `base:`. These _scopes_ can be selectively accessed by adding the name to the accessed endpoint, e.g. `curl http://localhost:8080/metrics/application` or `curl http://localhost:8080/metrics/base`.
====

== Add a few more

Let's add a few more metrics for our Kafka stream we setup in the previous exercise. Open the `NameConverter` class (in the `org.acme.people.stream` package), and add these metrics annotations to the `process()` method:

[source,java,role="copypaste"]
----
@Counted(name = "convertedNames", monotonic = true, description = "How many names have been converted.")
@Timed(name = "converter", description = "A measure how long it takes to convert names.", unit = MetricUnits.MILLISECONDS)
----

Don't forget to import the correct classes as before.

== Rebuild Executable JAR

Now we are ready to run our application on the cluster. Using the command palette, select **Create Executable JAR**. You should see a bunch of log output that ends with a `SUCCESS` message.

== Deploy

Let's deploy our app to the cluster and see if Prometheus picks up our metrics! To do this, start the container build using our executable JAR:

[source,sh,role="copypaste"]
----
oc start-build people --from-file target/*-runner.jar --follow
----

== Confirm deployment

Run and wait for the app to complete its rollout:

[source,sh,role="copypaste"]
----
oc rollout status -w dc/people
----

== Test

You'll need to trigger the methods that we've instrumented, so first run this command to get the URL to the word cloud page we previously created, and then open it in your browser, which will start producing names (and generating metrics):

[source,sh,role="copypaste"]
----
echo http://$(oc get route people -o=go-template --template='{{ .spec.host }}')/names.html
----

Within a few seconds, Prometheus should start scraping the metrics. Run this command to output the URL to the Prometheus GUI:

[source,sh,role="copypaste"]
----
echo http://$(oc get route prometheus -o=go-template --template='{{ .spec.host }}')
----

Open a separate browser tab and navigate to that URL. You should see:

::img

This is the Prometheus GUI which lets you issue queries to retrieve metrics Prometheus has gathered. Start typing in the box to look for 'acme':

image:prom.png[Prometheus,800]

These are the metrics exposed by our application, both raw numbers (like number of converted names in the `application:org_acme_people_stream_name_converter_converted_names` metric) along with quantiles of the same data across different time periods (e.g. `application:org_acme_people_stream_name_converter_converter_rate_per_second`).

Select `application:org_acme_people_stream_name_converter_converted_names` in the box, and click **Execute**. This will fetch the values from our metric showing the number of converted names:

image:promnames.png[names,800]

Click the **Graph** tab to see it visually, and adjust the time period to `5m`:

image:promg1.png[names,800]

Cool! You can try this with some of the JVM metrics as well, e.g. try to graph the `process_resident_memory_bytes` to see how much memory our app is using over time:

image:promg2.png[names,800]

Of course Quarkus apps use very little memory, even for apps stuffed with all sorts of extensions and code.

== Beyond Prometheus

Prometheus offers a powerful way to capture metrics, a rich query language and alerting of unusual or unexpected conditions. Another open source project, https://grafana.com/[Grafana] is commonly used as a flexible, graphical frontend which has support for Prometheus (and many other data sources) and can display https://prometheus.io/docs/visualization/grafana/[customized, realtime dashboards]:

image::https://grafana.com/api/dashboards/3308/images/2099/image[Grafana dashboard,800]

== Extra Credit after this workshop

If you have time after completing this workshop, try running grafana and hooking it up to your Prometheus instance you've created. Hint: `oc new-app grafana/grafana && oc expose svc/grafana` and then access Grafana at the URL emitted by this command: `echo http://$(oc get route grafana -o=go-template --template='{{ .spec.host }}')`, add Prometheus as a data source at `http://prometheus:9090`, and create a custom dashboard showing some of your application metrics, or import an existing dashboard from the Grafana community.

== Cleanup

Go to the first Terminal tab and press kbd:[CTRL+C] to stop our locally running app (or close the Terminal window).

== Congratulations!

This exercise demonstrates how your Quarkus application can utilize the https://github.com/eclipse/microprofile-metrics[MicroProfile Metrics] specification through the SmallRye Metrics extension.

There are many more possibilities for application metrics, and it's a useful way to not only gather metrics, but act on them through alerting and other features of the monitoring stack you may be using.