///////////////////////////////////////////////////////////////////////////////

    Copyright (c) 2018, 2020 Oracle and/or its affiliates.

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

        http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.

///////////////////////////////////////////////////////////////////////////////

:javadoc-base-url-api: {javadoc-base-url}io.helidon.config/io/helidon/config

= Extensions
:description: Helidon config extensions
:keywords: helidon, config

Developer-provided extensions influence how the config system behaves.

The <<config/01_introduction.adoc,config system introduction>> explains the design of the config
system and how its parts work together to read and parse config data, convert it
to Java types, fine-tune the look-up of config data, and reload and
reprocess data when it changes. _Config extensions_ provided by the application
modify and expand the way the config system performs these steps. 

== Introduction
Each config extension implements one of the interfaces defined in the Configuration SPI:

// TODO 
//Once our asciidoc processing handles labeled lists, uncomment the following
//and use it instead of the bulleted list which follows:
//`ConfigSource`:: Loads raw configuration data from a given type of source and
//delegates to a `ConfigParser`, producing the in-memory data structure which
//represents the loaded and parsed configuration.
//`ConfigParser`:: Translates configuration content in a given format into the
//corresponding internal config data structures.
//`OverrideSource`:: Provides key/value pairs which override config values loaded
//from any `ConfigSource`, given the key but _ignoring_ the original value.
//`ConfigFilter`:: Transforms config `String` values returned from any value-type
//`Config` node, given the key _and_ the original value.
//`ConfigMapperProvider`:: Provides one or more ``ConfigMapper``s each of which
//converts a `Config` object tree to a Java type specific to the application.
//`PollingStrategy`:: Implements a custom technique for notifying the Config system
//when the data underlying a `ConfigSource` or `OverrideSource` has changed.
* `ConfigSource` - Loads raw configuration data from a given type of source and delegates to a `ConfigParser`, producing the in-memory data structure which represents the loaded and parsed configuration.
* `ConfigParser` - Translates configuration content in a given format into the corresponding internal config data structures.
* `OverrideSource` - Provides key/value pairs which override config values loaded from any `ConfigSource`, given the key and _ignoring_ the original value.
* `ConfigFilter` - Transforms config `String` values returned from any value-type
`Config` node, given the key _and_ the original value.
* `ConfigMapperProvider` - Provides one or more ``ConfigMapper``s each of which converts a `Config` object tree to a Java type specific to the application.
* `PollingStrategy` - Implements a custom technique to trigger polling of underlying sources for changes
* `ChangeWatcher` - Implements a custom technique to watch underlying sources for changes and notifying the config system of such a change

The extension mechanism of Config can also use Java `ServiceLoader`.
For this purpose, you implement providers that serve as factories for your implementation of an extension.
This is to support meta-configuration even for custom extensions.
Service providers:

* `ConfigMapperProvider` - support for config mappers, automatically discovered by the config system
* `ConfigFilter` - support for config filters, automatically discovered by the config system
* `ConfigParser` - support for config parsers, automatically discovered by the config system
* `ConfigSourceProvider` - support for named config sources, configurable through meta configuration
* `ChangeWatcherProvider` - support for named change watchers, configurable through meta configuration
* `OverrideSourceProvider` - support for named override sources, configurable through meta configuration
* `PollingStrategyProvider` - support for named polling strategies, configurable through meta configuration
* `RetryPolicyProvider` - support for retry policies, configurable through meta configuration

The config system itself implements several of these SPIs, as noted in the sections below.

== Setting up an extension

You can configure a custom extension in two ways:

1. Manual configuration with builder
2. Automatic configuration using a Java service loader

=== Manual configuration with builder

The following example shows configuration of all possible extensions with `Config` (all custom extension have a name prefix `My`):

[source,java]
----
Config config = Config.builder()
                .addSource(FileConfigSource.builder()
                                   .changeWatcher(MyChangeWatcher.create())
                                   .pollingStrategy(MyPollingStrategy.create())
                                   .parser(MyConfigParser.create())
                                   .retryPolicy(MyRetryPolicy.create()))
                .addSource(MySource.create())
                .addFilter(MyFilter.create())
                .overrides(MyOverrides.create())
                .build()
----

=== Automatic configuration using a service loader

The following extensions are loaded using a service loader for any configuration instance, and do not require an explicit setup:

* `ConfigParser` - each config parser on the classpath that implements `ConfigParserProvider` as a Java service loader service
* `ConfigFilter` - each filter on the classpath that implements `ConfigFilter` as a Java service loader service

Other extensions are only used from Java service loader when you use meta configuration.
Mapping is done through the type configured in meta configuration, and the type defined by the extension provider interface.
For example for config sources, the interface defines the following methods (only subset shown):

[source,java]
----
boolean supports(String type);
ConfigSource create(String type, Config metaConfig);
----

Considering the following meta configuration:

[source,yaml]
----
sources:
  - type: "my-type"
    properties:
      my-config: "configuration"
----

The config system would iterate through all `ConfigSourceProvider` implementations found through Java `ServiceLoader` based on their priority.
First provider that returns `true` when `supports("my-type")` is called would be used, and an instance of a `ConfigSource` created using `create("my-type", config)`, where `config` is located on the node of `properties` from meta configuration.

=== About Priority [[priority-info]]

The config system invokes extensions of a given type in priority order.
Developers can express the relative importance of an extension by annotating the service implementation class with
`@javax.annotation.Priority`.
The default value is 100. A _lower_ priority value represents _greater_ importance.

== ConfigSource SPI [[Config-SPI-ConfigSource]]

The config system includes built-in support for several types of sources
(for example, Java `String`, `Readable`, `Properties`, and `Map`
objects - see link:{javadoc-base-url-api}/ConfigSources.html[`ConfigSources`]).
Implement a link:{javadoc-base-url-api}/spi/ConfigSource.html[`ConfigSource`] to
load raw configuration data from a type of source that the config system does
not already support.

[plantuml,config/spi-ConfigSource,png,title="ConfigSource SPI",align="center"]
----
together {
    hide empty members

    interface ConfigSource {
        + void init(ConfigContext context)
        ..
        + String description()
        + boolean exists()
        + boolean optional()
    }

    interface ConfigContext {
        + ConfigSourceRuntime sourceRuntime(ConfigSource source)
    }

    interface ParsableSource extends ConfigSource {
        + Optional<Content> load()
        + Optional<ConfigParser> parser()
        + Optoinal<String> mediaType()
    }

    interface NodeConfigSource extends ConfigSource {
        + Optional<NodeContent> load()
    }

    interface LazyConfigSource extends ConfigSource {
        + Optional<ConfigNode> node(String key)
    }

    interface PollableSource extends ConfigSource {
        + isModified(Object stamp)
        + Optional<PolllingStrategy> pollingStrategy()
    }

    interface WatchableSource extends ConfigSource {
        + Class<T> targetType();
        + T target()
        + Optional<ChangeWatcher<T>> changeWatcher()
    }

    interface EventConfigSource extends ConfigSource {
        + void onChange(BiConsumer<String, ConfigNode> changedNode);
    }
}

ConfigSource -[hidden]> ConfigContext
----

For config sources that work directly with config nodes, the followin API is available.
These interfaces have an implementation provided by Helidon.
The interfaces `ConfigNode`, `ObjectNode`, `ValueNode` and
`ListNode` represent the in-memory data structure for loaded and parsed configuration data.

[plantuml,config/spi-node,png,title="ConfigNode API",align="center"]
----
hide empty members

interface ConfigNode {
    Optional<String> value()
}

interface ConfigNode.ValueNode extends ConfigNode {
    {static} + ValueNode create(String value)
    + String get()
}

interface ConfigNode.ListNode extends ConfigNode {
    {static} + Builder builder()
}
note bottom of ConfigNode.ListNode: extends List<ConfigNode>

interface ConfigNode.ObjectNode extends ConfigNode {
    {static} + ObjectNode empty()
    {static} + ObjectNdoe simple(String key, String value)
    {static} + Builder builder()
}
note bottom of ConfigNode.ObjectNode: extends Map<String, ConfigNode>
----

For config sources that work return data (`NodeConfigSource` and `ParsableConfigSource`) a
`Content` must be returned that describes the loaded data.
The following diagram depicts the `Content` API.

[plantuml,config/spi-content,png,title="Content API",align="center"]
----
hide empty members

interface ConfigContent {
    + void close()
    + Optional<Object> stamp()
}

interface NodeContent extends ConfigContent {
    + ConfigNode.ObjectNode data()
}

interface ConfigParser.Content extends ConfigContent {
    + Optional<String> mediaType()
    + Charset charset()
    + InputStream data()
}
----

Some of the methods provided are not always mandatory, yet they are part of the APIs to simplify the overall class structure:

* ConfigContent.stamp() - this method is used by `PollingStrategy` to determine if content has been changed.
This can be always
`empty` for sources, that do not implement `PollableSource`
* ConfigParser.Content.charset() - this can return any `Charset` for media types that are binary
* ConfigParser.Content.mediaType() - this can be used to override media type (that would otherwise be "guessed" from the underlying source)
* ParsableSource.parser() - this can be used to override parser (that would otherwise be based on `mediaType`)
* ParsableSource.mediaType() - return the configured or "guessed" media type of this source, see
`io.helidon.common.media.type.MediaTypes`, if not returned, media type must be present on `Content`, or provided through media type mapping

== ConfigParser SPI [[Config-SPI-ConfigParser]]

The parsing step converts config data in some format into the corresponding in-memory representation of config ``ObjectNode``s.
The config system can already parse several data formats (for example Java `Properties`, YAML, and HOCON).
Implement the
link:{javadoc-base-url-api}/spi/ConfigParser.html[`ConfigParser`] SPI to allow the config system to handle additional formats.

[plantuml,config/spi-ConfigParser,png,title="ConfigParser SPI",align="center"]
----
hide empty members

interface ConfigParser {
    + Set<String> getSupportedMediaTypes()
    + ObjectNode parse(Content content)
}

interface ConfigParser.Content {
    + Optional<String> mediaType()
    + Charset charset()
    + InputStream data()
}

ConfigParser -[hidden]> ConfigParser.Content
----

The `ConfigParser.Content` interface defines operations on the content that is to be parsed by a `ConfigParser` implementation:

* `mediaType()` - Reports the media type of the content (if it is to override media type defined on the config source)
* `data()` - Provides the `InputStream` with config source data
* `charset()` - Defines the charset to use to parse the stream in case this is a text based media type, ignored by parsers of binary content


The application can register parsers for a builder by invoking `Config.Builder#addParser(ConfigParser)`.
The config system also uses the Java service loader mechanism to load automatically, for all builders, any parsers listed in the
`META-INF/services/io.helidon.config.spi.ConfigParser` resource on the runtime classpath.
Prevent autoloading of parsers for a given builder by invoking `Config.Builder#disableParserServices()`.

`ConfigParser` accepts `@Priority`.
See <<priority-info, About Priority>>.

[source,java]
.Example custom parser implementation listed in `META-INF/services/io.helidon.config.spi.ConfigParser`
----
my.module.MyConfigParser
----

[source,java]
.Example custom parser definition in `module-info.java`
----
module my.module {
    requires transitive io.helidon.config;
    provides io.helidon.config.spi.ConfigParser with myModule.MyConfigParser;
}
----

== OverrideSource SPI [[Config-SPI-OverrideSource]]

When the application retrieves a configuration value the config system first uses
the relevant config sources and filters. It then applies any _overrides_ the
application has provided. Each override has:

* a `Predicate<Config.Key>` (a boolean-valued function that operates on 
the config key), and 
* a replacement, _overriding_, `String` value the config system should use if the predicate evaluates to true.

To furnish overrides to the config system, implement the
link:{javadoc-base-url-api}/spi/OverrideSource.html[`OverrideSource`] SPI one or more times and pass instances of those implementations to the config builder's
link:{javadoc-base-url-api}/Config.Builder.html#overrides-java.util.function.Supplier-[`overrides`]
method.
The config system will apply the overrides returned from each
`OverrideSource` to each config key requested from a `Config` that is based on that `Config.Builder`.

To support custom override sources in meta configuration, also implement the
link:{javadoc-base-url-api}/spi/OverrideSourceProvider.html[`OverrideSourceProvider`] service loader SPI

[plantuml,config/spi-OverrideSource,png,title="OverrideSource SPI",align="center"]
----
interface OverrideSource {
    + Optional<OverrideSource.OverrideData> load() throws ConfigException
}
----

Note that override sources can also implement `PollableSource`, and `WatchableSource` to add change support.

== ConfigFilter SPI [[Config-SPI-ConfigFilter]]

Before returning a `String` from `Config.value()` the config system applies any _filters_ set up on the `Config.Builder` used to create the config tree that contains the config node of interest.
The application provides filters as implementations of the
link:{javadoc-base-url-api}/spi/ConfigFilter.html[`ConfigFilter`] interface.
Each filter is a function which accepts a `Config.Key` and an input `String` value and returns a `String` value the config system should use for that key going forward.
The filter can return the original value or return some other value. 

The application registers filters and filter providers by passing `ConfigFilter` 
implementations to one of the config builder
link:{javadoc-base-url-api}/Config.Builder.html[`addFilter` methods]. The config 
system also uses the Java service loader mechanism to load 
additional filters automatically, for all builders, using 
the service interface described in the following table. Prevent a given 
builder from using the auto-loaded filters by invoking the
link:{javadoc-base-url-api}/Config.Builder.html#disableFilterServices--[`disableFilterServices`]
method.

.Config SPI Interfaces for Filtering
|===
|Interface |Method |Usage

|link:{javadoc-base-url-api}/spi/ConfigFilter.html[`ConfigFilter`]

Accepts `@Priority`. See <<priority-info, About Priority>>.
|`String apply(Config.Key key, String stringValue);`
|Accepts a key and the corresponding `String` value and 
returns the `String` which the config system should use for that key.
|===

=== Initializing Filters
The `ConfigFilter` JavaDoc describes multiple methods for adding filters to a
`Config.Builder`. Some accept a `ConfigFilter` directly and some accept a provider 
function which, when passed a `Config` instance, returns a `ConfigFilter`.

*_Neither a `ConfigFilter` nor a provider function which furnishes one should 
access the `Config` instance passed to the provider function._*

Instead, implement the `ConfigFilter.init(Config)` method on the filter. The config
system invokes the filters' `init` methods according to the filters' `@Priority` 
order. 

Recall that whenever any code invokes `Config.get`, the `Config` instance
invokes the `apply` method of _all_ registered filters. By the time the application
retrieves config this way the config system will have run the `init` method on all
the filters. _But note that when a filter's `init` method invokes `Config.get`, the 
`init` methods of lower-priority filters will not yet have run._

[plantuml, config/spi-ConfigFilter, png, title="ConfigFilter SPI", align="center"]
----
interface ConfigFilter {
    + String apply(Config.Key key, String stringValue)
}
----

== ConfigMapperProvider SPI [[Config-SPI-ConfigMapperProvider]]

The config system provides built-in mappings from `String` values to various Java 
types. (See link:{javadoc-base-url-api}/ConfigMappers.html[`ConfigMappers`].) 

To handle mappings to other types the application can register
custom mappers with the config system by implementing the 
link:{javadoc-base-url-api}/spi/ConfigMapperProvider.html[`ConfigMapperProvider`]
 SPI.

Such providers return a map, with entries in which:

* the key is the Java type (a `Class` object) the mapper produces, and
* the value is a `ConfigMapper` that converts the config in-memory
data structure into the type in the key.

The provider may also implement other methods for finer tuned conversion mechanisms:

* `genericTypeMappers()` returns a map with entries for specific `GenericType` conversions,
    for example when the provider supports only mapping for `GenericType<Map<String, Integer>>`
* `mapper(Class)` returns a conversion function (optional) that converts a config node
    to the typed instance (if supported by this provider)
* `mapper(GenericType)` returns a conversion function (optional) that coverts a config node
    to the GenericType (if supported by this provider) - for example in case this provider supports
    any Map<String, ?> type, such as `Map<String, Integer>` and `Map<String, Double>`

The config conversion system works as follows:

For `Config.as(Class)`:

1. Check whether a conversion function exists for the class requested (from method `mappers()`).
2. Check whether a conversion function is provided by any `ConfigMapperProvider` with method `mapper(Class)`.
3. Check whether a conversion function exists for a generic type for the class requested (from method `genericTypeMappers`).
4. Check whether a conversion function is provided by any `ConfigMapperProvider` with method `mapper(GenericType)` for
    a generic type for the class requested.

For `Config.as(GenericType)` - the first two steps are skipped.

The config system also uses the Java `ServiceLoader` mechanism to load automatically,
for all builders, any mappers returned by the providers listed in the 
`META-INF/services/io.helidon.config.spi.ConfigMapperProvider` resource on the
runtime classpath. The application can prevent autoloading of mappers for a
given builder by invoking `Config.Builder#disableMapperServices()`. Note
that the built-in mappers described in `ConfigMappers` still operate.

Mapper providers accept `@Priority`. See <<priority-info, About Priority>>. 

[plantuml, config/spi-ConfigMapperProvider, png, title="ConfigMapperProvider SPI", align="center"]
----
interface ConfigMapperProvider {
    + Map<Class<?>, Function<Config, ?>> getMappers()
    + Map<GenericType<?>, BiFunction<Config, ConfigMapper, ?>> genericTypeMappers()
    + <T> Optional<Function<Config, T>> mapper(Class<T> type)
    + <T> Optional<BiFunction<Config, ConfigMapper, T>> mapper(GenericType<T> type)
}
----


A mapper provider can specify a `@javax.annotation.Priority`.
If no priority is explicitly assigned, the value of `100` is assumed.

[source,java]
.Reference custom mapper provider implementation in `META-INF/services/io.helidon.config.spi.ConfigMapperProvider`
----
my.module.MyConfigMapperProvider
----

[source,java]
.Reference custom mapper provider implementation in `module-info.java`
----
module my.module {
    requires transitive io.helidon.config;
    provides io.helidon.config.spi.ConfigMapperProvider with my.module.MyConfigMapperProvider;
}
----

== Change support SPI [[Config-SPI-PollingStrategy]]

Once it loads a `Config` tree from ``ConfigSource``s the config system does not itself change the in-memory `Config` tree.
Even so, the underlying data available via the tree's ``ConfigSource``s can change.
Implementations of link:{javadoc-base-url-api}/spi/PollingStrategy.html[`PollingStrategy`]
may trigger regular check whether a source has new data.
Implementation of link:{javadoc-base-url-api}/spi/ChangeWatcher.html[`ChangeWatcher`]
may watch the underlying source for changes and trigger an update.

=== PollingStrategy SPI

An implementation of `PollingStrategy` gets an instance to poll, and triggers its `poll`
method.
The result of `poll` method may be used to update the polling strategy schedule.

The approach of checking for changes is part of the config system, and the `PollingStrategy` does not need to be concerned with it.
This is based on the source `stamp` as defined in `ConfigContent`
and used in `PollableSource.isModified(Object)` methods.

If a more sophisticated solution is needed, you may need to implement a `ChangeWatcher` instead.

The config system offers polling strategy for periodic time-based checks.
Often an application can create a config source simply by using one of the methods on `ConfigSources` (for example,
`ConfigSources#file(path)` to get a builder and then invoke `pollingStrategy`
passing a polling strategy.
But the application can implement its own `PollingStrategy` and set it on the config source builder instead.

[plantuml,config/spi-PollingStrategy,png,title="PollingStrategy SPI",align="center"]
----
hide empty members

interface PollingStrategy {
    + void start(Polled polled)
    + stop()
}

interface PollingStrategy.Polled {
    + ChangeEventType poll(Instant when)
}
----

To support polling strategies that can be configured in meta configuration, also implement the `PollingStrategyProvider` Java service loader SPI.

=== ChangeWatcher SPI

An implementation of `ChangeWatcher` gets the underlying source information and a change listener.
The "watcher" then watches for changes of the source and notifies the listener when a change occurs.

This is designed to support sources that can react on changes (such as file system).
When a polling mechanism is needed, please check `PollingStrategy` above.

The config system offers a change watcher for any `Path` based config source (such as `FileConfigSource`) and for the `etcd` config source.

To use a change watcher, simply create a config source using its builder and register the change watcher on the builder (the config source must support appropriate type of change watchers).

[plantuml,config/spi-ChangeWatcher,png,title="ChangeWatcher SPI",align="center"]
----
hide empty members

interface ChangeWatcher<T> {
    + void start(T target, Consumer<ChangeEvent<T>> listener)
    + stop()
    + Class<T> type()
}

interface ChangeWatcher.ChangeEvvent<T> {
    + Instant changeTime()
    + T target()
    + ChangeEventType type()
    {static} + create(T target, ChangeEventType type, Instant instant)
    {static} + create(T target, ChangeEventType type)
}
----

To support change watchers that can be configured in meta configuration, also implement the `ChangeWatcherProvider` Java service loader SPI.

== RetryPolicy SPI [[Config-SPI-RetryPolicy]]

The builder for each `ConfigSource` and `OverrideSource` accepts a 
link:{javadoc-base-url-api}/spi/RetryPolicy.html[`RetryPolicy`] 
governing if and how the source should deal with failures loading the underlying 
data. 

A retry policy accepts a function, the invocation of which the policy will
govern according to its own implementation. 
Applications can use the predefined policies in 
link:{javadoc-base-url-api}/RetryPolicies.html[`RetryPolicies`], such as 
`RetryPolicies.justCall` which simply invokes the function without any retry.
That class also exposes a builder for constructing a time-based retry policy, 
with several parameters:

.Parameters Controlling Built-in `RetryPolicy`
|===
|Parameter |Usage |Default

|`delay` |Initial delay between calls to the function | 200 ms
|`delayFactor` |Multiplier applied to `delay` on each successive call | 2
|`callTimeout` |Time limit for each individual call of the function | 500 ms
|`overallTimeout` |Limit for the total elapsed time attempting to 
call the function successfully, including delays between calls | 2 s
|===

The actual delay between function call starts as `delay` and changes by the factor
`delayFactor` on each successive attempt.

Note that the job of each retry policy is to call the provided function 
successfully. As such, the policy must perform the first attempt as well
as any retries.

[plantuml, config/spi-RetryPolicy, png, title="RetryPolicy SPI", align="center"]
----
interface RetryPolicy {
    + <T> T execute(Supplier<T> call)
    + boolean cancel(boolean mayInterruptIfRunning)
}
----

The application can try to cancel the overall execution of a `RetryPolicy` by invoking the `RetryPolicy#cancel(boolean mayInterruptIfRunning)` method.
Ideally the retry policy implementation should be able to abort the execution of the retry policy, even while a function call is in progress, but the policy must respond to cancels between function calls.
In either case `cancel` returns `true` if the retry was aborted without a successful call to the function, and `false` otherwise, including if the function call had already completed successfully or had previously been successfully canceled.

To support retry policies in meta configuration, also implement the Java service loader SPI
`RetryPolicyProvider`.

