///////////////////////////////////////////////////////////////////////////////

    Copyright (c) 2019 Oracle and/or its affiliates. All rights reserved.

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.

///////////////////////////////////////////////////////////////////////////////

= Kubernetes Probes
:description: Kubernetes probes
:keywords: helidon, readiness, liveness, probes, kubernetes
:kubernetes-probes-url: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes

This document describes how to use the Helidon health check API with Kubernetes.

== About Kubernetes probes

Probes is the term used by Kubernetes to describe health checks for containers
 (link:{kubernetes-probes-url}[Kubernetes documentation]).

There are two types of probes:

* `liveness`: Indicates whether the container is running
* `readiness`: Indicates whether the container is ready to service requests

You can implement probes using the following mechanisms:

. Running a command inside a container
. Sending an `HTTP` request to a container
. Opening a `TCP` socket to a container

A microservice exposed to HTTP traffic will typically implement both the
 liveness probe and the readiness probe using HTTP requests.

You can configure several parameters for probes. The following are the most
 relevant parameters:

[cols="2,5",role="flex, sm7"]
|=======
| `initialDelaySeconds`
| Number of seconds after the container has started before liveness or readiness
 probes are initiated.

| `periodSeconds`
| Probe interval. Default to 10 seconds. Minimum value is 1.

| `timeoutSeconds`
| Number of seconds after which the probe times out. Defaults to 1 second.
 Minimum value is 1

| `failureThreshold`
| Number of consecutive failures after which the probe should stop. Default: 3.
 Minimum: 1.
|=======

=== Liveness probe

The liveness probe is used to verify the container has become unresponsive.
 For example, it can be used to detect deadlocks or analyze heap usage. When
 Kubernetes gives up on a liveness probe, the corresponding pod is restarted.

NOTE: The liveness probe can result in repeated restarts in certain cases.
 For example, if the probe is implemented to check all the dependencies
 strictly, then it can fail repeatedly for temporary issues. Repeated restarts
 can also occur if `timeoutSeconds` or `periodSeconds` is too low.

We recommend the following:

* Avoid checking dependencies in a liveness probe.
* Set `timeoutSeconds` to avoid excessive probe failures.
* Acknowledge startup times with `initialDelaySeconds`.

=== Readiness probe

The readiness probe is used to avoid routing requests to the pod until it is
 ready to accept traffic. When Kubernetes gives up on a readiness probe, the
 pod is not restarted, traffic is not routed to the pod anymore.

NOTE: In certain cases, the readiness probe can cause all the pods to be removed
 from service routing. For example, if the probe is implemented to check all the
 dependencies strictly, then it can fail repeatedly for temporary issues. This
 issue can also occur if `timeoutSeconds` or `periodSeconds` is too low.

We recommend the following:

* Be conservative when checking shared dependencies.
* Be aggressive when checking local dependencies.
* Set `failureThreshold` according to `periodSeconds` in order to accommodate
 temporary errors.

== Troubleshooting probes

Failed probes are recorded as events associated with their corresponding pods.
 The event message contains only the status code.

[source,bash]
.Get the events of a single pod:
----
POD_NAME=$(kubectl get pod -l app=acme -o jsonpath='{.items[0].metadata.name}') # <1>
kubectl get event --field-selector involvedObject.name=${POD_NAME} # <2>
----
<1> Get the effective pod name by filtering pods with the label `app=acme`.
<2> Filter the events for the pod.

TIP: Create log messages in your health check implementation when setting a
 `DOWN` status. This will allow you to correlate the cause of a failed probe.

== Example

This example shows the usage of the Helidon health API in an application that
 implements health endpoints for the liveness and readiness probes. Note that
 the application code dissociates the health endpoints from the default routes,
 so that the health endpoints are not exposed by the service. An example YAML
 specification is also provided for the Kubernetes service and deployment.

[source,java]
.Application code:
----
Routing healthRouting = Routing.builder()
        .register(JsonSupport.create())
        .register(HealthSupport.builder()
                .webContext("/live") // <1>
                .add(HealthChecks.healthChecks()) // <2>
                .build())
        .register(HealthSupport.builder()
                .webContext("/ready") // <3>
                .add(() -> HealthCheckResponse.named("database").up().build()) // <4>
                .build())
        .build();

Routing defaultRouting = Routing.builder()
        .any((req, res) -> res.send("It works!")) // <5>
        .build();

WebServer server = WebServer.builder(defaultRouting)
        .config(ServerConfiguration.builder()
                .port(8080) // <6>
                .addSocket("health", SocketConfiguration.builder() // <7>
                        .port(8081)
                        .build())
                .build())
        .addNamedRouting("health", healthRouting) // <8>
        .build();

server.start();
----
<1> The health service for the `liveness` probe is exposed at `/live`.
<2> Using the built-in health checks for the `liveness` probe.
<3> The health service for the `readiness` probe is exposed at `/ready`.
<4> Using a custom health check for a pseudo database that is always `UP`.
<5> The default route: returns It works! for any request.
<6> The server uses port 8080 for the default routes.
<7> A socket configuration named `health` using port `8081`.
<8> Route the health services exclusively on the `health` socket.

[source,yaml]
.Kubernetes descriptor:
----
kind: Service
apiVersion: v1
metadata:
  name: acme # <1>
  labels:
    app: acme
spec:
  type: NodePort
  selector:
    app: acme
  ports:
  - port: 8080
    targetPort: 8080
    name: http
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: acme # <2>
spec:
  replicas: 1
  template:
    metadata:
      name: acme
      labels:
        name: acme
    spec:
      containers:
      - name: acme
        image: acme
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /live # <3>
            port: 8081
          initialDelaySeconds: 3 # <4>
          periodSeconds: 10
          timeoutSeconds: 3
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /ready # <5>
            port: 8081
          initialDelaySeconds: 10 # <6>
          periodSeconds: 30
          timeoutSeconds: 10
---
----
<1> A service of type `NodePort` that serves the default routes on port `8080`.
<2> A deployment with one replica of a pod.
<3> The HTTP endpoint for the liveness probe.
<4> The liveness probe configuration.
<5> The HTTP endpoint for the readiness probe.
<6> The readiness probe configuration.