///////////////////////////////////////////////////////////////////////////////

    Copyright (c) 2018, 2019 Oracle and/or its affiliates. All rights reserved.

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.

///////////////////////////////////////////////////////////////////////////////

:javadoc-base-url-api: {javadoc-base-url}?io/helidon/config

= Advanced Configuration Topics
:description: Helidon config advanced configuration
:keywords: helidon, config
:toc: preamble
:toclevels: 4

This section discusses several advanced topics related to Helidon configuration.

== Advanced Config Sources and Config Parsers
=== Environment Variables Config Source

The config system supports using environment variables as a config source, and is
enabled by default. Since environment variable names are normally restricted to
alphanumeric characters and underscore, this config source _adds_ aliases that
enable setting or overriding config entries with dotted and/or hyphenated keys.

The mapping makes it possible to set or override a config entry with a key of
`"foo.bar"` using an environment variable named `"FOO_BAR"` and `"foo.bar-baz"`
using `"FOO_BAR_dash_BAZ"`.


One use case for this mapping is config overrides in containers, where passing
environment variables directly or via Kubernetes Secrets/ConfigMaps is common.
Scripts that solve the mapping problem by explicitly converting variables to
system properties can also be simplified.

Aliases are produced for any environment variable name that matches _all_ of
the following:

<1> does not begin or end with a `'_'` character
<2> does not contain `"__"`
<3> contains one or more `'_'` characters

For each such name, two aliases are added with the names mapped as follows:

<1> Replace any `"\_dash_"` or `"\_DASH_"` substrings with `"-"`, e.g. `"APP_PAGE_dash_SIZE"`
becomes `"APP_PAGE-SIZE"`.

<2> Replace `'_'` with `'.'` and add as an alias, e.g. `"APP_GREETING"` is
added as `"APP.GREETING"` and `"APP_PAGE-SIZE"` is added as `"APP.PAGE-SIZE"`.
This mapping is added primarily to support mixed case config keys such as
`"app.someCamelCaseKey"`.

<3> Convert the result of step 2 to lowercase and add as an alias, e.g.
`"APP.GREETING"` is added as `"app.greeting"` and `"APP.PAGE-SIZE"` is added
as `"app.page-size"`.

=== Directory Config Source

The config system supports using a file system directory as a config source.
Each _non-directory_ file in the directory becomes a config entry: the file name
is the key and the contents of that file
are used as the corresponding config `String` value.

The following example shows, for example, one way to load Kubernetes secrets
mounted on the pod's filesystem.

If the directory `conf/secrets` contains these two files

[source]
.File `secrets/username`
----
jose
----

[source]
.File `secrets/password`
----
^ery$ecretP&ssword
----

your application can load this as configuration as follows:

[source,java]
.Using `directory` config source
----
Config secrets = Config.withSources(
        ConfigSources.directory("conf/secrets")) // <1>
        .disableEnvironmentVariablesSource()     // <2>
        .disableSystemPropertiesSource()         // <2>
        .build();

assert secrets.get("username")                   // <3>
        .asString()
        .get()
        .equals("jose");
assert secrets.get("password")                   // <4>
        .asString()
        .get()
        .equals("^ery$ecretP&ssword");
----

<1> Loads all files from the `conf/secrets` directory.
<2> No need to use environment variables or system properties as sources in building
 the `Config`.
<3> The loaded config maps the key `username` to the value `jose`...
<4> ...and the key `password` to `^ery$ecretP&ssword`.

Remember that your application can process the contents of a given file
as configuration. See the <<config/02_config-sources.adoc,config sources>> section
and the link:{javadoc-base-url-api}/ConfigSources.html#file-java.lang.String-[`ConfigSources.file`]
JavaDoc.

=== In-memory Config Sources
The config system provides several ways to create a `Config` tree from data
already in memory. See the link:{javadoc-base-url-api}/ConfigSources.html[`ConfigSources` javadoc]
for further details. The numerous variants of the `from` method construct
`ConfigSource` or `Builder<ConfigSource>` instances.

==== Subtree of Another `Config`
[source,java]
----
Config anotherConfig = Config.create(classpath("application.conf"));

Config config = Config.create(
        ConfigSources.create(anotherConfig.get("data")));
----
==== `Properties` Object
[source,java]
----
Config config = Config.create(
        ConfigSources.create(System.getProperties()).build());                // <1>
----

==== `String` of a Given Media Type
[source,java]
----
Config config = Config.create(
        ConfigSources.create("app.greeting = Hi", "text/x-java-properties")); // <2>
----
==== `Map`
[source,java]
----
Config config = Config.crate(
        ConfigSources.create(Map.of("app.page-size", "20"))
            .lax()             // <3>
            .build());         // <1>
----
==== _ad hoc_ Config Nodes
[source,java]
----
Config config = Config.create(
        ConfigSources.create(ObjectNode.builder()
                                   .addList("app.basic-range", ListNode.builder()
                                           .addValue("-20")
                                           .addValue("20")
                                           .build())
                                   .build()));
----
<1> `ConfigSources.create` variants for `Properties` or `Map` arguments return a
 `ConfigSources.MapBuilder` instance.
<2> A similar `create` variant accepts a `Readable` instead of a `String`.
<3> `MapBuilder` by default throws an exception if a key appears more than once
in the map. The `lax()` method relaxes this; the config system logs a warning instead.

=== Multi-Source ``Config``s and Composite Config Sources
Although the examples above use a single source, you can build a single `Config`
from multiple sources.

==== Handling Key Collisions
===== Prefixed Config Sources
Sometimes you might want to create a single config tree from
multiple sources but in a way that keeps the config from different sources
in different subtrees.

The config system lets you assign a prefix to all keys
from a given source using the
link:{javadoc-base-url-api}/ConfigSources.html#prefixed-java.lang.String-java.util.function.Supplier-[`ConfigSources.prefixed`] method.
The following example shows two YAML files as config sources
and the code to load each with a different prefix into a single `Config` tree:

[source,hocon]
.File `app.conf`
----
greeting = "Hello"
page-size = 20
basic-range = [ -20, 20 ]

----

[source,hocon]
.File `data.conf`
----
providers: [
    {
        name = "Provider1"
        class = "this.is.my.Provider1"
    },
    {
        name = "Provider2"
        class = "this.is.my.Provider2"
    }
]

----

[source,java]
.Using `prefixed` config source
----
Config config = Config.create(
        ConfigSources.prefixed("app",                    // <1>
                               classpath("app.conf")),   // <2>
        ConfigSources.prefixed("data",                   // <3>
                               classpath("data.conf"))); // <4>

assert config.get("app.greeting")                        // <5>
        .asString()
        .get()
        .equals("Hello");

assert config.get("data.providers.0.name")               // <6>
        .asString()
        .get()
        .equals("Provider1");
----

<1> Specifies the prefix `app` for the associated source.
<2> `Supplier<ConfigSource>` for the file
 `app.conf` loaded from the current `classpath`.
<3> Specifies the prefix `data` for the associated source.
<4> Supplier<ConfigSource> for the file `app.conf` loaded from the current `classpath`.
<5> Key `app.greeting` combines the `app` prefix and the original key `greeting`
from the `app.conf` source.
<6> Key `data.providers.0.name` combines the `data` prefix and
 the original key `providers.0.name` property from `data.conf` source.

This technique can be useful, for example, if multiple sources contain
keys that might overlap; assigning different prefixes to the keys from different
sources gives your application a way to access all config elements distinctly even
if their keys would otherwise conflict.

===== Merging Strategies
The `ConfigSources.create(Supplier<ConfigSource>...)` and `ConfigSources.create(List<Supplier<ConfigSource>...)`
methods return a `CompositeBuilder`.
By default, earlier sources in the list have higher priority than later ones, meaning
that if the same key appears in two or more sources the source earlier in the
list prevails.

Each ``CompositeConfigSource``'s _merging strategy_ actually controls this behavior.
The config system provides the
`FallbackMergingStrategy`
which implements the default, "first wins" algorithm. You can write your own
implementation of
link:{javadoc-base-url-api}/ConfigSources.MergingStrategy.html[`ConfigSources.MergingStrategy`]
and use it instead to provide a different algorithm.

[source,java]
.Composite config source example
----
Config config = Config.create(                                               // <1>
        ConfigSources.create(file("conf/dev.properties").optional(),         // <2>
                           file("conf/config.properties").optional())        // <2>
                .add(classpath("application.properties"))                    // <3>
                .mergingStrategy(ConfigSources.MergingStrategy.fallback())); // <4>
----

<1> Creates a new `Config` instance from a single composite config source.
<2> Method `ConfigSources.create(sources...)` returns `CompositeBuilder` instance
 initialized with two sources (from `dev.properties` and `config.properties`
 files).
<3> Adds third config source (`application.properties` on
 classpath) to the same `CompositeBuilder`.
<4> Specifies the merging strategy. This example uses the default fallback
 merging strategy.


=== How Config Chooses Parsers [[Config-Advanced-Sources-SuitableParser]]
As the link:config/02_config-sources.html[config sources and parsers] section describes,
these two work together to read and translate configuration data from some
external form into the corresponding in-memory config tree.

Although most applications are
explicit about the config sources they use in building a `Config`, the config system often
has to figure out what parser to use. It does so by:

1. determining, the best that it can, the media type of the source, and
2. locating a parser that can translate that media type.

==== Identifying the Media Type

===== By Inference
Most applications let the config system try to infer the media type of the
config source.

By default config source implementations use the Java Service
`java.nio.file.spi.FileTypeDetector` API to infer the source media type from
the source, typically (but not always) based on the file type portion of the file path.
The config system registers implementations of
the `java.nio.file.spi.FileTypeDetector` SPI that recognize
the supported formats: `.properties`, `.yaml`, `.json` and `.conf`. To handle
other formats you can implement and register your own `FileTypeDetector`
implementations. (Typically you would also write and register a config parser
to translate that format; see <<locating-parser,Locating a Parser>> below.)

===== By Application Directive
Your application can specify what media type to use in interpreting a config
source. Use this if your application knows the media type but the system might
not be able to infer it correctly, either because no type detector would recognize it or
because there might be more than one inferred media type.

[source,java]
.Specify `mediaType` for config source
----
Config config = Config.create(classpath("props")                             // <1>
                                    .mediaType("text/x-java-properties")); // <2>
----

<1> The config system cannot infer the media type because there is no file
type in the path `props`.
<2> The developer knows the file is in Java Properties format so specifies the
media type explicitly.

Note that a file type detector _could_ be written to
also inspect the contents of the file to infer the media type. The detector
which the config system provides looks only for the `.properties` file type in the
path.

==== Locating a Parser [[locating-parser]]
===== By Inference from `media-type`
Each config parser reports which media types it handles. Once the config system
has determined a source's media type, it searches the config parsers associated
with the config builder for one that recognizes that media type. It then uses
that parser to translate the config in the source into the in-memory config tree.

The application can add one or more parsers to a `Config.Builder`
using the `addParser` method. This makes the parser available for use by the
config sources associated with that builder, but does not directly tie a given
parser to a given source. The builder uses media-type matching to select one of
the parsers registered with the builder for each source.

If the config system cannot locate a parser that matches the media type of a source, it throws
a `ConfigException` when trying to prepare the configuration.

===== By Application Directive
Your application can specify which parser to use for a config source. The
`AbstractParsableConfigSource.Builder` class exposes the `parser` method, which
accepts the `ConfigParser` to be used for that source. Several methods
on `ConfigSources` such as `classpath`, `directory`, and `file` return this
builder class.

Generally try to rely on media-type matching rather than specifying a given parser
for a given source in the application. This keeps your application more flexible,
both by insulating it from implementation classes and by letting it easily take
advantage of improvements in or alternatives to the parsers available for a given
media type.

[source,java]
.Specify `parser` for config source
----
Config config = Config.create(classpath("props")                            // <1>
                                    .parser(ConfigParsers.properties())); // <2>
----

<1> The config system cannot infer the media type because there is no file
type in the path `props`.
<2> The developer knows the file is in Java Properties format so specifies the
properties parser explicitly.

=== Parsing a Config Value as Config
A config value node might contain an entire config document in `String` form, but in
a format different from the containing document. Your application can tell the
config system to parse such a node as config in a different format and replace
the `String` value node in the original tree with the config tree that results
from parsing that `String`.

In this example, a YAML document contains a JSON document as a leaf.

[source,yaml]
.YAML file with included JSON formated property
----
secrets:
    username: "jose"
    password: "^ery$ecretP&ssword"

app: >                             # <1>
    {
        "greeting": "Hello",
        "page-size": 20,
        "basic-range": [ -20, 20 ]
    }

----

<1> The property `app` is itself formatted as a JSON document.

==== Specify Key-to-media-type Mapping
[source,java]
.Specify JSON as media type for node
----
Config config = Config.create(
        classpath("application.yaml")
                .mediaTypeMapping(                          // <1>
                        key -> "app".equals(key.toString()) // <2>
                                ? "application/json"
                                : null));

assert config.get("secrets.username").asString()            // <3>
        .get().equals("jose");
assert config.get("secrets.password").asString()            // <3>
        .get().equals("^ery$ecretP&ssword");

assert config.get("app").type() == Type.OBJECT;             // <4>

assert config.get("app.greeting")                           // <3>
        .asString().get().equals("Hello");
assert config.get("app.page-size")                          // <3>
        .asInt().get() == 20;
assert config.get("app.basic-range.0")                      // <3>
        .asInt().get() == -20;
assert config.get("app.basic-range.1")                      // <3>
        .asInt().get() == 20;
----

<1> The source builder's `mediaTypeMapping` method accepts a function
which returns the appropriate media types (if any) for config keys.
<2> The function says to treat the `app` property value as a JSON document and
leave other nodes unchanged.
<3> Other properties are loaded as expected.
<4> Property `app` is now an structured object node.

Because the function passed to `mediaTypeMapping` identifies the `app` node as a JSON
document, the config system selects the config parser that is registered with the builder
which also handles the JSON media type.

Also, note that the config system replaces the original `String` value node with
an object node resulting from parsing that `String` value as JSON.

==== Specify Key-to-parser Mapping
Alternatively, your application could map config keys to the specific parsers
you want to use for parsing those keys' values.

[source,java]
.Specify JSON formatted property' parser instance
----
Config config = Config.create(
        ConfigSources.classpath("application.yaml")
                .parserMapping(                                           // <1>
                        key -> "app".equals(key.toString())               // <2>
                                ? HoconConfigParserBuilder.buildDefault()
                                : null));
----

<1> Uses the `parserMapping` method to map keys to parser instances.
<2> Tells the config system to use the HOCON parser for translating the `String`
value of the `app` key. (HCON is a superset of JSON.)

As before, the config system replaces the value node in the
containing config tree with the config tree resulting from the additional parse.

=== Loading Config using Meta-configuration [[Config-Advanced-metaconfig]]
Instead of including code in your application to construct config trees from
builders, sources, etc., you can instead prepare _meta-configuration_ in a file
that declares the sources to load and their attributes.

You can either specify the meta-config file in your application or
allow the config system to search for and load meta-config from a preset
list of possible sources.

==== Loading Config by Specifying a Meta-configuration File [[Config-Advanced-Sources-MetaSource]]
Your application loads
the configuration specified by a meta-config file by:

* invoking the link:{javadoc-base-url-api}/ConfigSources.html#load-io.helidon.config.Config-[`ConfigSources.load(Config)`]
method, passing a config object read from the meta-config source as the argument;
* invoking the link:{javadoc-base-url-api}/Config.html#loadSourcesFrom-java.util.function.Supplier...[`Config.loadSourcesFrom`]
method, or
* invoking the link:{javadoc-base-url-api}/Config.html#builderLoadSourcesFrom-java.util.function.Supplier...[`Config.builderLoadSourceFrom`]
method.

These methods return either a `Config` tree or a `Config.Builder` which
your application can further fine-tune before using to construct a `Config`
tree. The config system interprets the meta-config as directions for how to
build a config tree, rather than as the config data itself.

==== Loading Config from an Implicit Meta-configuration File [[Config-Advanced-Config-MetaConfig]]
The <<config/01_introduction.adoc,introduction>> section shows how to use
`Config.create()` to load config from one of several possible default config files.
That same method also searches for one of several possible default meta-config files
from which to load config sources to be used for the default config.

The `Config.create()` method determines the default configuration from
the following search:

. Attempt to load _meta-config_ from at most one of the following, checked in this order:
.. `meta-config.yaml` - meta configuration file in YAML format
.. `meta-config.conf` - meta configuration file in HOCON format
.. `meta-config.json` - meta configuration file in JSON format
.. `meta-config.properties` - meta configuration file in Java Properties format
. Otherwise, load _config_ from:
.. environment variables, and
.. Java system properties, and
.. at most one of the following, checking in this order:
... `application.yaml` - configuration file in YAML format
... `application.conf` - configuration file in HOCON format
... `application.json` - configuration file in JSON format
... `application.properties` - configuration file in Java Properties format

Remember that the config system will check for these default meta-config and config files
only if the classpath includes the corresponding parsers. The introduction section on <<config/01_introduction.adoc#built-in-formats,built-in formats>>
section describes this further.

==== Meta-configuration File Format
Each meta-configuration file must contain the top-level `sources` property that is an
array (ordered list) of config sources. The meta-config file can contain other
top-level keys as well but the config system ignores them when it interprets the contents as
meta-configuration.

Each `sources` property must contain exactly one of following top level properties:

.Meta-configuration Required Top-level Property
|===
|Property Name |Usage

|`type` a|Either: +

* a predefined type (see <<MetaConfig-built-in-types,below>>), or
* a custom config source ID
|`class` a|Fully-qualified class name of either: +

* a custom config source implementation, or
* a builder class with a `build()` method that returns `ConfigSource`
|===

If you specify both `type` and `class`, the config system ignores the `class`
setting.

In addition, each `sources` property can optionally have a `properties` property
which assigns type-specific attributes for the config source being defined.

===== Built-in Types [[MetaConfig-built-in-types]]
The config system supports these built-in types:

.Built-in Meta-configuration Types
|===
|Type |Use |Related `ConfigSources` Method |Required Properties

|`system-properties` |System properties are a config source |`ConfigSources.systemProperties()` | n/a
|`environment-variables` |Environment variables are a config source |`ConfigSources.environmentVariables()` | n/a
|`classpath` |Specified resource is used as a config source |`ConfigSources.classpath(String)` | `resource` - path to the resource to load
|`file` |Specified file is used as a config source |`ConfigSources.file(Path)` |`path` - path to the file to load
|`directory` |Each file in directory used as config entry, with key = file name and value = file contents |`ConfigSources.directory(String)` |`path` - path to the directory to use
|`url` |Specified URL is read as a config source |`ConfigSources.url(URL)` | `url` - URL from which to load the config
|`prefixed` |Associated config source is loaded with the specified prefix |`ConfigSources.prefixed(String,Supplier)` a|* `key` - key of config element in associated source to load
* `type` or `class` - associated config source specification
* `properties` - as needed to further qualify the associated config source
|===

Except for the `system-properties` and `environment-variables` types, the meta-config
`properties` section for a source can also specify any optional settings for the
corresponding config source type. The JavaDoc for the related config source
type builders lists the supported properties for each type. (For example,
link:{javadoc-base-url-api}/internal/FileConfigSource.FileBuilder.html[`FileConfigSource.FileBuilder`].)

Here is example meta-configuration in HOCON format. Note how the `properties` sections
are at the same level as the `type` or `class` within a `sources` array entry.

[source,hocon]
.Meta-configuration `config-meta-all.conf` illustrating all built-in sources available on the classpath
----
sources = [
    {
        type = "environment-variables"
    }
    {
        type = "system-properties"
    }
    {
        type = "directory"
        properties {
            path = "conf/secrets"
            media-type-mapping {
                yaml = "application/x-yaml"
                password = "application/base64"
            }
            polling-strategy {
                type = "regular"
                properties {
                    interval = "PT15S"
                }
            }
        }
    }
    {
        type = "url"
        properties {
            url = "http://config-service/my-config"
            media-type = "application/hocon"
            optional = true
            retry-policy {
                type = "repeat"
                properties {
                    retries = 3
                }
            }
        }
    }
    {
        type = "file"
        properties {
            path = "conf/env.yaml"
            polling-strategy {
                type = "watch"
            }
        }
    }
    {
        type = "prefixed"
        properties {
            key = "app"
            type = "classpath"
            properties {
                resource = "app.conf"
            }
        }
    }
    {
        type = "classpath"
        properties {
            resource = "application.conf"
        }
    }
]
----

Note that the example shows how your meta-configuration can configure optional features such as polling
strategies and retry policies for config sources.

==== Meta-config for Custom Source Types
You can use meta-config to set up custom config source types as well as the
built-in ones described above. Meta-config supports this in two ways:

* by class name
* by custom type name

===== Custom Source Types using `class`
Use the `class` property in one of your `sources` entries and as its value
give the fully-qualified class name of your custom source type. The config
system will use that class as the `ConfigSource` (or as a builder for one)
for that source.

[source,hocon]
----
{
    class = "io.helidon.config.git.GitConfigSourceBuilder"
    properties {
        path = "application.conf"
        directory = "/app-config"
    }
}
----

===== Custom Source Type using `type`
You can add your own custom type names to the built-in ones by adding to a
`META-INF/resources/meta-config-sources.properties` file on the classpath.
In this file each property name is a custom config source type name and its
value is the fully-qualified class name for a custom `ConfigSource` implementation
or a builder for it.

For example, the Helidon module `helidon-config-git` provides this
`META-INF/resources/meta-config-sources.properties` file:

[source]
.Definition of `git` Config Source Type
----
git = io.helidon.config.git.GitConfigSourceBuilder
----

This definition lets you configure a git config source in meta-configuration as
follows:

[source,hocon]
----
{
    type = "git"
    properties {
        path = "application.conf"
        directory = "/app-config"
    }
}
----

You can define and use your own custom config source type names similarly.

Note that it is the `AbstractSource` SPI class that provides support for
polling strategies and retry policies. If you create custom config sources that
should also offer this support be
sure they extend `AbstractSource` or one of its subclasses to inherit this behavior.

==== Loading Config using Meta-configuration
Here is how your application can use meta-configuration in a particular resource on the
classpath to load a `Config` tree:

[source,java]
.Loading Config using Meta-configuration
----
ConfigSource sourceFromMetaConfig = ConfigSources.load(  // <1>
        classpath("config-meta-all.conf")).build();      // <2>

Config config = Config.create(sourceFromMetaConfig);       // <3>
----
<1> The `ConfigSources.load` method creates a config source for the eventual config
from the config source argument which specifies the meta-config.
<2> This example uses meta-config from a resource on the classpath but you can
use meta-config from any valid config source.
<3> The `load` method populates the `sourceFromMetaConfig` `ConfigSource` from
all the actual sources declared in the meta-configuration. The returned `ConfigSource`
is ready for use in creating a `Config` instance.

==== Meta-config for Polling Strategies and Retry Policies

Your meta-config can include the set-up for polling strategies and retry
policies if the config source supports them. Declare them in a way similar to
how you declare the config sources themselves: by `type` or `class` and with
accompanying `properties`.

.Meta-config Support for Built-in Polling Strategies
|===
|Strategy Type |Usage |Properties

|`regular`
| Periodic polling - See link:{javadoc-base-url-api}/PollingStrategies.html#regular-java.time.Duration-[`PollingStrategies.regular`] method
|`interval` (`Duration`) - indicating how often to poll; e.g., `PT15S` represents 15 seconds

|`watch`
| Filesystem monitoring - See link:{javadoc-base-url-api}/PollingStrategies.html#watch-java.nio.file.Path-[`PollingStrategies.watch`] method
| `path` - file system path to the `classpath`, `file`, or `directory` to monitor

|===

.Meta-config Support for Built-in Retry Policies
|===
|Policy Type |Usage |Properties

|`repeat`
|Regularly-scheduled - see link:{javadoc-base-url-api}RetryPolicies.html#repeat-int-[`RetryPolicies.repeat`].
a|`retries` (`int`) - number of retries to perform +

Optional:

* `delay` (`Duration`) - initial delay between retries
* `delay-factor` (`double`) - `delay` is repeatedly multiplied by this each retry to compute
the delay for each successive retry
* `call-timeout` (`Duration`) - timeout for a single invocation to load the source
* `overall-timeout` (`Duration`) - total timeout for all retry calls and delays
|===

To specify a custom polling strategy or custom retry policy, specify `class` (instead of
`type`) and give the fully-qualified class name for the implementation class.
If your custom class needs parameters to control its behavior the config system
uses `io.helidon.config.ConfigMapper` to initialize the class instance.
See link:{javadoc-base-url-api}/spi/RetryPolicy.html[`RetryPolicy`] and
link:{javadoc-base-url-api}/spi/PollingStrategy.html[`PollingStrategy`] JavaDoc
sections.

== Configuration Key
As described in the <<config/03_hierarchical-features.adoc#accessByKey,hierarchical features
section>> each config node (except the root) has a non-null key. Here is the formal
definition of what keys can be:
[source,abnf]
.The ABNF syntax of config key
----
config-key = *1( key-token *( "." key-token ) )
 key-token = *( unescaped / escaped )
 unescaped = %x00-2D / %x2F-7D / %x7F-10FFFF
           ; %x2E ('.') and %x7E ('~') are excluded from 'unescaped'
   escaped = "~" ( "0" / "1" )
           ; representing '~' and '.', respectively
----

[IMPORTANT]
=========
To emphasize, the dot character ("`.`") has special meaning as a name separator
in keys. To include a dot as a character in a key escape it as
 "`~1`". To include a tilda escape it as "`~0`".
=========

For example, the following configuration file contains two object nodes with
names `oracle` and `oracle.com`.

[source,json]
.Example `application.json` with dot character in key
----
{
    "oracle" : {
        "com" : true,
        "cz" : false
    },
    "oracle.com" : {
        "secured" : true
    }
}

----

[source,java]
.Working with configuration with dot character in node name
----
Config config = Config.create(classpath("application.json"));

// node `oracle`
assert config.get("oracle.com").asBoolean().get() == true;                   // <1>
assert config.get("oracle").get("com").asBoolean().get() == true;            // <1>
assert config.get("oracle.com").type() == Type.VALUE;                        // <2>
assert config.get("oracle.com").name().equals("com");                        // <3>
// node `oracle.com`
assert config.get("oracle~1com.secured").asBoolean().get() == true;          // <4>
assert config.get(Key.escapeName("oracle.com"))                              // <5>
        .get("secured").asBoolean().get() == true;
assert config.get(Key.escapeName("oracle.com")).type() == Type.OBJECT;       // <6>
assert config.get(Key.escapeName("oracle.com")).name().equals("oracle.com"); // <7>
----

<1> Work with the first `oracle` object as usual.
As always you can use the fully-qualified key `oracle.com` or chain `get(key)`
 calls to access the `com` property value.
<2> Config node `"oracle"` / `"com"` is a leaf node (has type `VALUE`)...
<3> ... and has the name `com` (the last token in its key).
<4> The second object has name `oracle.com`. The code must escape the
dot in the node's name using `oracle~1com`.
<5> Or, use the utility method `Config.Key.escapeName(name)` to escape dots or tildes
that might be in the node's name, in this example in `oracle.com`.
<6> The config node `"oracle.com"` has type `OBJECT`...
<7> ...and name `"oracle.com"`.


== Filter, Overrides, and Token Substitution [[filters-and-overrides]]
When your application retrieves a config value, the config system can transform it
before returning the value, according to _filters_, _overrides_, and
_tokens_. The config system provides some built-in instances of these
you can use, and you can add your own as described in the
sections which describe
<<config/07_extensions.adoc#Config-SPI-ConfigFilter,filters>> and
<<config/07_extensions.adoc#Config-SPI-OverrideSource,overrides>>.

Your application can add filters and overrides explicitly to a config builder
and the config system by default uses the Java service loader mechanism to
locate all available filters and overrides and add them automatically to all
config builders (unless your code disables that behavior for a given
builder).

=== Filters
Each filter accepts a key and the value as defined in the source, and returns
the value to be used. The filter can leave the value unchanged or
alter it, as it sees fit.

The built-in link:{javadoc-base-url-api}/ConfigFilters.html#valueResolving--[value-resolving]
filter enables the token substitution described below.

See the link:{javadoc-base-url-api}/spi/ConfigFilter.html[`ConfigFilter`] JavaDoc
for more information.

=== Overrides
The overrides feature allows you to create an external document containing key/value
pairs which replace the value otherwise returned for the name, and then add that
document as an override source to a config builder.

There are some key differences between overrides and filters.

* Because overrides are loaded
from sources those sources can change while your application runs and so the
overrides they that prescribe can change.
* The override document can use wildcards in key expressions.
* Overrides can affect only keys that already exist in the original source; filters
can supply values even if the key is absent from the config source.

Each override entry consists of a Java properties-format definition. The key is an
expression (which can use wildcards) to match config keys read from the current
config sources, and the override value is the new value for any key matching the
key expression from that entry. Order is important. The
config system tests every key expression/value pair one by one in the order they appear
in the overrides sources. Once the config system finds an override entry in which
the key expression matches the configuration key, the system returns that entry's
value for the key being processed.

See the link:{javadoc-base-url-api}/spi/OverrideSource.html[`OverrideSource]` JavaDoc
for more detail.

=== Tokens
A token reference is a key token starting with `$`, optionally enclosed between
 `{` and `}`, i.e. `$ref`, `${ref}`. Even a key composed of more than one token
 can be referenced in another key, i.e. `${env.ref}`.

As an example use case, you can use token references to declare the default values (see
`resolving-tokens.yaml` below), while the references may be resolved in another
config source, which identifies a current environment (see `env.yaml` examples
below). You can then use the same overrides for different environments, say `test` and `prod`.
The configuration in each environment is then overridden with a different values
using wildcards (see `overrides.properties` below).

[source,java]
.Initialize `Config` with Override Definition from `overrides.properties` file
----
Config config = Config.builder()
        .overrides(OverrideSources.file("conf/overrides.properties")) // <1>
        .sources(file("conf/env.yaml"),                               // <2>
                 classpath("resolving-tokens.yaml"))                  // <3>
        .build();
----

<1> Loads _overrides_ from the specified file.
<2> A deployment-specific environment configuration file.
<3> A default configuration containing token references that are resolved
using the environment-specific override.

== Executors for Asynchronous Config Activity
Various parts of the config system work asychronously:

* polling strategies to detect changes to config sources,
* publishers to notify your application when such changes occur,
* `Config` instances which subscribe to and respond to change notifications for
their underlying sources, and
* retry policies (which might wait between retries).

Each of these uses an executor to perform its work. The config system provides default
executors, but your application can specify different ones if necessary.

=== Executors for Polling Strategy
The two methods `PollingStrategies.regular(Duration)` and
`PollingStrategies.watch(Path)` return builders for their respective strategies.
Both builders expose the `executor` method which your application can invoke, passing a
`java.util.concurrent.ScheduledExecutorService` instance it wants used for the
polling work. By default each polling strategy instance uses a separate thread
pool executor.

The following example shares the same executor for two different polling
strategy instances.
[source,java]
.Customize polling strategy executors
----
ScheduledExecutorService executor = Executors.newScheduledThreadPool(2); // <1>

Config config = Config.create(
        ConfigSources.file("conf/dev.properties")
                .pollingStrategy(
                        PollingStrategies.regular(Duration.ofSeconds(2)) // <2>
                                .executor(executor)),                    // <3>
        ConfigSources.create("conf/config.properties")
                .pollingStrategy(
                        path -> PollingStrategies.watch(path)            // <4>
                                .executor(executor)));                   // <5>
----

<1> Prepares a thread pool executor with core pool size set `2` to be shared by
 all polling strategies.
<2> Selects the built-in periodic polling strategy.
<3> Tells the config system to use the specific executor to poll the
`dev.properties` config source.
<4> Uses the Java filesystem `WatchService` to monitor the specified path.
<5> Tells the config system to use the same executor to monitor the path.

=== Publishers for Source Change Events
Recall that when a polling strategy detects a change in a source, it informs
interested parties of the changes. By default each `Config.Builder` arranges
for the resulting `Config` tree to use a shared executor that reuses available threads
from a pool, creating new threads as needed. The same executor is used for actually
reloading the source.

Your application can invoke the polling strategy builder's `changesExecutor` method to
tell the builder
to use a different `Executor`. (As an aside, your application can also control
the size of the buffer used for holding source change events by invoking the
builder's `changesMaxBuffer` method. The default is 256.)

[source,java]
.Customize config and override sources' executors
----
Executor executor = Executors.newCachedThreadPool();               // <1>

Config config = Config.builder()
        .overrides(
                OverrideSources.file("conf/overrides.properties")
                        .pollingStrategy(PollingStrategies::watch)
                        .changesExecutor(executor)                 // <2>
                        .changesMaxBuffer(4))                      // <3>
        .sources(
                ConfigSources.file("conf/env.yaml")
                        .pollingStrategy(PollingStrategies::watch)
                        .changesExecutor(executor)                 // <4>
                        .changesMaxBuffer(4))                      // <4>
        .build();
----

<1> Prepares a thread pool executor to be shared by selected sources.
<2> Tells the builder that the resulting overrides source should use the specified
`Executor` for notifying interested parties of changes and for reloading the
override source.
<3> Specifies an event buffer size of 4.
<4> Uses the same `Executor` and event buffer size for the config source as for
the override source above.

=== Composite Config Source Executor
When your application supplies multiple sources to a config builder, as with
`Config.create(Supplier<ConfigSource>...)` and `Config.create(List<Supplier<ConfigSource>>)`,
the config system
automatically uses a _composite config source_ which aggregates the separate
sources but also listens for changes to any of the individual sources so it can
delegate the change notification. For this change detection and notification the
config system, by default, uses an executor with a dedicated thread pool that is
shared across all `Config` instances.

Your application can invoke the builder's `changesExecutor` method to use a
different `ScheduledExecutorService` instance.
The builder returned by the `from` methods mentioned above is a
link:{javadoc-base-url-api}/ConfigSources.CompositeBuilder.html[CompositeBuilder]
which extends `Config.Builder`.

Because a composite source might yield more numerous change events -- because of the
multiple underlying sources -- your application can specify a _debounce timeout_
for the composite source by invoking the `CompositeBuilder.changesDebounce(Duration)`
method. The composite source aggregates multiple change events within this period
into a single event and broadcasts that one instead and reloads the sources at
that time, not necessarily in response to every single change in any source.
The default is `100` milliseconds.

[source,java]
.Customize composite source executors
----
ScheduledExecutorService executor = Executors.newScheduledThreadPool(1);       // <1>

Config config = Config.create(
        ConfigSources.create(file("conf/dev.properties")                         // <2>
                                   .pollingStrategy(PollingStrategies::watch),
                           file("conf/config.properties")                      // <2>
                                   .pollingStrategy(PollingStrategies::watch))
                .changesExecutor(executor)                                     // <3>
                .changesMaxBuffer(4)                                           // <4>
                .changesDebounce(Duration.ofSeconds(1)));                      // <5>
----

<1> Prepares a thread pool executor.
<2> `ConfigSources.create(Supplier<ConfigSource>...)` creates and returns a
`CompositeBuilder` based on the two sources.
<3> Specifies a particular executor for monitoring and change event notification.
<4> Sets the subscriber's buffer size to 4 events. The composite source discards
any events not consumed by a subscriber if it needs to create room for more
recent events.
<5> Change events will not fire more frequently than once per a second.

=== Config Custom Executor
A loaded config tree subscribes to change events publishes by its source(s).
By default, each `Config` uses an executor which manages a dedicated thread pool
reusing previously-created threads when they are available and creating new threads
as needed.
All `Config` instances share the dedicated thread pool.

Your application
can specify a non-default `Executor` for a tree to use for accepting and propagating
those events by invoking the `changesExecutor` method on the `Config.Builder`.
Each source subscriber has a dedicated buffer for holding changes events. This
defaults to 256 but you can tailor this value as needed.

[source,java]
.Customize config executor
----
Executor executor = Executors.newCachedThreadPool();        // <1>

Config config = Config.create(
        file("conf/config.properties")
                .pollingStrategy(PollingStrategies::watch))
        .changesExecutor(executor)                          // <2>
        .changesMaxBuffer(16)                               // <3>
        .build();
----

<1> Prepares a specific thread pool executor.
<2> Specifies the executor the `Config` tree will use to listen for and propagate
change events.
<3> Sets the event subscriber buffer to `16` events.

=== Retry Policy Custom Executor
You can control which executor a retry policy should use for its work.
The `RetryPolicies.repeat(int retries)` method returns
a link:{javadoc-base-url-api}/RetryPolicies.Builder.html[RetryPolicies.Builder].
Your application can invoke the retry policy builder's `executor` method to
specify which `ScheduledExecutorService` instance it should use to schedule and
execute delayed retries. By default the config system uses a separate thread
pool executor for each retry policy instance.

[source,java]
.Customize retry policy executors
----
ScheduledExecutorService executor = Executors.newScheduledThreadPool(2, myThreadFactory); // <1>

Config config = Config.create(
        ConfigSources.file("conf/dev.properties")
                .optional()                                                               // <2>
                .retryPolicy(RetryPolicies.repeat(2)                                      // <3>
                        .executor(executor)));                                            // <4>
----

<1> Prepares a thread pool executor with core pool size set to `2` and a custom
 `java.util.concurrent.ThreadFactory`.
<2> When the source is flagged as `optional()`, the loading attempt will be
 repeated as the retry policy defines, but an overall failure will _not_ lead to
failing the initial load or preventing the source from being polled if so configured.
<3> Uses the built-in _repeating_ implementation of `RetryPolicy` that can be used with any
 config source, but typically for ones that might suffer brief, intermittent outages.
<4> Specifies the executor to use for loading and retries.
