Skip to content

Commit

Permalink
Support custom ILM aliases (elastic#3826)
Browse files Browse the repository at this point in the history 
* Refactor ILM config loading and conditional index selection

* Make rollover_alias configurable and document it

closes elastic#3779
  • Loading branch information
@simitt
simitt authored May 29, 2020
1 parent ad56c42 commit 4c7f516
Show file tree
Hide file tree
Showing 23 changed files with 718 additions and 677 deletions.
18 changes: 14 additions & 4 deletions _meta/beat.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -382,17 +382,25 @@ apm-server:
# Default value is `true`.
#require_policy: true

# The configured event types and policies will be merged with the default setup. You only need to configure
# the mappings that you want to customize.
# The configured event types, policies and rollover_aliases will be merged with
# the default setup. Only configure the mappings that should be customized.
# NOTICE: when customizing `rollover_alias`, it is strongly recommended to keep
# the prefix "apm-%{[observer.version]}" unchanged;
# in case the prefix is changed, `setup.template.name` and `setup.template.pattern`
# need to be changed accordingly to match the created indices
#mapping:
#- event_type: "error"
# policy_name: "apm-rollover-30-days"
# rollover_alias: "apm-%{[observer.version]}-error"
#- event_type: "span"
# policy_name: "apm-rollover-30-days"
# rollover_alias: "apm-%{[observer.version]}-span"
#- event_type: "transaction"
# policy_name: "apm-rollover-30-days"
# rollover_alias: "apm-%{[observer.version]}-transaction"
#- event_type: "metric"
# policy_name: "apm-rollover-30-days"
# rollover_alias: "apm-%{[observer.version]}-metric"

# Configured policies are added to pre-defined default policies.
# If a policy with the same name as a default policy is configured, the configured policy overwrites the default policy.
Expand Down Expand Up @@ -485,12 +493,14 @@ apm-server:
#setup.template.enabled: true

# Template name. By default the template name is "apm-%{[observer.version]}"
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
# The template name and pattern has to be set in case the elasticsearch index pattern is modified,
# or the `apm-server.ilm.setup.mapping` rollover_aliases are changed.
#setup.template.name: "apm-%{[observer.version]}"

# Template pattern. By default the template pattern is "apm-%{[observer.version]}-*" to apply to the default index settings.
# The first part is the version of apm-server and then -* is used to match all daily indices.
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
# The template name and pattern has to be set in case the elasticsearch index pattern is modified,
# or the `apm-server.ilm.setup.mapping` rollover_aliases are changed.
#setup.template.pattern: "apm-%{[observer.version]}-*"

# Path to fields.yml file to generate the template.
Expand Down
18 changes: 14 additions & 4 deletions apm-server.docker.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -382,17 +382,25 @@ apm-server:
# Default value is `true`.
#require_policy: true

# The configured event types and policies will be merged with the default setup. You only need to configure
# the mappings that you want to customize.
# The configured event types, policies and rollover_aliases will be merged with
# the default setup. Only configure the mappings that should be customized.
# NOTICE: when customizing `rollover_alias`, it is strongly recommended to keep
# the prefix "apm-%{[observer.version]}" unchanged;
# in case the prefix is changed, `setup.template.name` and `setup.template.pattern`
# need to be changed accordingly to match the created indices
#mapping:
#- event_type: "error"
# policy_name: "apm-rollover-30-days"
# rollover_alias: "apm-%{[observer.version]}-error"
#- event_type: "span"
# policy_name: "apm-rollover-30-days"
# rollover_alias: "apm-%{[observer.version]}-span"
#- event_type: "transaction"
# policy_name: "apm-rollover-30-days"
# rollover_alias: "apm-%{[observer.version]}-transaction"
#- event_type: "metric"
# policy_name: "apm-rollover-30-days"
# rollover_alias: "apm-%{[observer.version]}-metric"

# Configured policies are added to pre-defined default policies.
# If a policy with the same name as a default policy is configured, the configured policy overwrites the default policy.
Expand Down Expand Up @@ -485,12 +493,14 @@ apm-server:
#setup.template.enabled: true

# Template name. By default the template name is "apm-%{[observer.version]}"
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
# The template name and pattern has to be set in case the elasticsearch index pattern is modified,
# or the `apm-server.ilm.setup.mapping` rollover_aliases are changed.
#setup.template.name: "apm-%{[observer.version]}"

# Template pattern. By default the template pattern is "apm-%{[observer.version]}-*" to apply to the default index settings.
# The first part is the version of apm-server and then -* is used to match all daily indices.
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
# The template name and pattern has to be set in case the elasticsearch index pattern is modified,
# or the `apm-server.ilm.setup.mapping` rollover_aliases are changed.
#setup.template.pattern: "apm-%{[observer.version]}-*"

# Path to fields.yml file to generate the template.
Expand Down
18 changes: 14 additions & 4 deletions apm-server.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -382,17 +382,25 @@ apm-server:
# Default value is `true`.
#require_policy: true

# The configured event types and policies will be merged with the default setup. You only need to configure
# the mappings that you want to customize.
# The configured event types, policies and rollover_aliases will be merged with
# the default setup. Only configure the mappings that should be customized.
# NOTICE: when customizing `rollover_alias`, it is strongly recommended to keep
# the prefix "apm-%{[observer.version]}" unchanged;
# in case the prefix is changed, `setup.template.name` and `setup.template.pattern`
# need to be changed accordingly to match the created indices
#mapping:
#- event_type: "error"
# policy_name: "apm-rollover-30-days"
# rollover_alias: "apm-%{[observer.version]}-error"
#- event_type: "span"
# policy_name: "apm-rollover-30-days"
# rollover_alias: "apm-%{[observer.version]}-span"
#- event_type: "transaction"
# policy_name: "apm-rollover-30-days"
# rollover_alias: "apm-%{[observer.version]}-transaction"
#- event_type: "metric"
# policy_name: "apm-rollover-30-days"
# rollover_alias: "apm-%{[observer.version]}-metric"

# Configured policies are added to pre-defined default policies.
# If a policy with the same name as a default policy is configured, the configured policy overwrites the default policy.
Expand Down Expand Up @@ -485,12 +493,14 @@ apm-server:
#setup.template.enabled: true

# Template name. By default the template name is "apm-%{[observer.version]}"
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
# The template name and pattern has to be set in case the elasticsearch index pattern is modified,
# or the `apm-server.ilm.setup.mapping` rollover_aliases are changed.
#setup.template.name: "apm-%{[observer.version]}"

# Template pattern. By default the template pattern is "apm-%{[observer.version]}-*" to apply to the default index settings.
# The first part is the version of apm-server and then -* is used to match all daily indices.
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
# The template name and pattern has to be set in case the elasticsearch index pattern is modified,
# or the `apm-server.ilm.setup.mapping` rollover_aliases are changed.
#setup.template.pattern: "apm-%{[observer.version]}-*"

# Path to fields.yml file to generate the template.
Expand Down
1 change: 1 addition & 0 deletions changelogs/head.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,4 @@ https://github.com/elastic/apm-server/compare/7.8\...master[View commits]
[float]
==== Added
* Support configurable response headers for the RUM endpoints {pull}3820[3820]
* Support custom ILM rollover aliases {pull}3826[3826]
76 changes: 43 additions & 33 deletions docs/ilm-reference.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ feature in {es} allows you to automate the lifecycle of your APM Server indices
IMPORTANT: ILM policies are only applied to indices created after the policies have taken effect.
In other words, any indices present before ILM was configured still need to be managed manually.

To learn how to define your own index lifecycle policies, see <<ilm,customize index lifecycle management>>.
To learn how to define your own index lifecycle policies or rollover aliases,
see <<ilm,customize index lifecycle management>>.

[float]
[[ilm-default]]
Expand Down Expand Up @@ -50,29 +51,33 @@ apm-server:
mapping:
- event_type: "error"
policy_name: "apm-rollover-30-days"
rollover_alias: "apm-%{[observer.version]}-error"
- event_type: "span"
policy_name: "apm-rollover-30-days"
rollover_alias: "apm-%{[observer.version]}-span"
- event_type: "transaction"
policy_name: "apm-rollover-30-days"
rollover_alias: "apm-%{[observer.version]}-transaction"
- event_type: "metric"
policy_name: "apm-rollover-30-days"
policies:
rollover_alias: "apm-%{[observer.version]}-metric"
policies:
- name: "apm-rollover-30-days"
policy:
phases:
hot:
actions:
rollover:
max_size: "50gb"
max_age: "30d"
set_priority:
priority: 100
warm:
min_age: "30d"
actions:
set_priority:
priority: 50
readonly: {}
policy:
phases:
hot:
actions:
rollover:
max_size: "50gb"
max_age: "30d"
set_priority:
priority: 100
warm:
min_age: "30d"
actions:
set_priority:
priority: 50
readonly: {}
----

[float]
Expand Down Expand Up @@ -109,37 +114,38 @@ _Example:_ You’re customizing a policy that’s already been sent to Elasticse
<<ilm-setup-config,`ilm.setup.enabled:true`>>
<<ilm-setup-overwrite-config,`ilm.setup.overwrite:true`>>

2+|*Disable ILM and Enable ILM setup.*
2+|*Enable ILM and Disable ILM setup.*

|_Details:_
Create unmanaged indices. Ensure event specific templates are set up accordingly.
Use managed indices. No policies, rollover aliases and templates will be created by APM Server.

_Example:_ You're creating your policies, rollover aliases and templates elsewhere, e.g., with Elasticsearch APIs.

|_Configure these options:_

<<ilm-enabled-config,`ilm.enabled:false`>>
<<ilm-setup-config,`ilm.setup.enabled:true`>>
<<ilm-setup-overwrite-config,`ilm.setup.overwrite:true`>>
<<ilm-enabled-config,`ilm.enabled:true`>>
<<ilm-setup-config,`ilm.setup.enabled:false`>>

2+|*Enable ILM and Disable ILM setup.*
2+|*Disable ILM to create unmanaged default indices.*

|_Details:_
Use managed indices. No policies and templates will be created by APM Server.

_Example:_ You're creating your policies and templates elsewhere, e.g., with Elasticsearch APIs.
Disable ILM to create unmanaged indices with default names. Ensure event specific templates are set up accordingly.

|_Configure these options:_

<<ilm-enabled-config,`ilm.enabled:true`>>
<<ilm-setup-config,`ilm.setup.enabled:false`>>
<<ilm-enabled-config,`ilm.enabled:false`>>
<<ilm-setup-config,`ilm.setup.enabled:true`>>
<<ilm-setup-overwrite-config,`ilm.setup.overwrite:true`>>

2+|*Use custom Indices*
2+|*Create unmanaged custom Indices*

|_Details:_
Configure event specific templates when using custom index names.
Disable ILM to create unmanaged indices with custom names. Ensure event specific templates are set up accordingly.

|_Configure these options:_

<<ilm-enabled-config,`ilm.enabled:auto`>>
<<ilm-setup-config,`ilm.setup.enabled:true`>>
<<ilm-setup-overwrite-config,`ilm.setup.overwrite:true`>>
`output.elasticsearch.index: "my-custom-index"`
|=======================================================================
Expand Down Expand Up @@ -207,7 +213,11 @@ APM Server will still make use of ILM and connect your template with the defined
[[ilm-setup-mapping-config]]
===== `apm-server.ilm.setup.mapping`

Maps each event type to the named policy. APM event types can only be `error`, `span`, `transaction`, and `metric`.
Policies defined must be mapped to an event type. If they are not, they will not be sent to Elasticsearch.
Maps each event type to a corresponding `policy_name` and `rollover_alias`.
APM event types can only be `error`, `span`, `transaction`, and `metric`.
If you attempt to map an index lifecycle policy to a different event type, APM Server will not start.
If you only map a subset of APM event types, the default values will be used for omitted event types.
If you only map a subset of APM event types, the default values will be used for omitted event types.
If a policy is defined, it must be mapped to an event type to take effect.
When customizing `rollover_alias`, it is strongly recommended to keep the prefix `apm-%{[observer.version]}` unchanged;
if the prefix must be changed, `setup.template.name` and `setup.template.pattern` need to be changed accordingly,
to match the created indices.
52 changes: 52 additions & 0 deletions docs/ilm.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ All of this can be done directly from the `apm-server.yml` file.

* <<ilm-enable>>
* <<ilm-setup>>
* <<custom-ilm-alias>>
* <<custom-ilm-policy>>
* <<example-ilm-config>>
* <<map-ilm-policy>>
Expand Down Expand Up @@ -59,6 +60,51 @@ apm-server:

ILM can now be customized via the remaining `apm-server.ilm.setup.*` configuration options.

[float]
[[custom-ilm-alias]]
==== Create a custom ILM `rollover_alias`

You can define a custom rollover alias per event type. The rollover alias can only contain a
restricted set of variables, limited to variables concerning `observer.*`. In case other
variables are configured, the server will refuse to start.
The related ILM policy can specify a rollover action, e.g. when an index reaches a size of 50GB
it should roll over to create a new index. The `rollover_alias` abstracts away the concrete write index, while
automatically being updated and keeping track of the current write index.
This allows to use the `rollover_alias` instead of specific indices in queries.

For example when defining a rollover alias with name `apm-7.8.0-transaction`, an alias pointing to a write index
`apm-7.8.0-transaction-000001` is created. When the index reaches the defined size or age it will roll over to
a new index `apm-7.8.0-transaction-000002`. The rollover alias `apm-7.8.0-transaction` keeps
track of which index is the write index when ingesting data.

NOTE: Ensure that the configured `rollover_alias` for all event types start with the same prefix,
and that this prefix is also configured for `setup.template.name` and `setup.template.pattern`.
If the template configuration is not accordingly changed, the Elasticsearch index template with
the predefined mappings will not match against created indices, leading to indexing issues.

The example below shows how to change the `rollover_alias` to a custom value,
and set the template configuration to a matching name and pattern.

[source,yml]
----
apm-server:
ilm:
enabled: "auto"
setup:
mapping:
- event_type: "error"
rollover_alias: "apm-dev-error"
- event_type: "span"
rollover_alias: "apm-dev-span"
- event_type: "transaction"
rollover_alias: "apm-dev-transaction"
- event_type: "metric"
rollover_alias: "apm-dev-metric"
setup.template.name: "apm-dev"
setup.template.pattern: "apm-dev*"
----

[float]
[[custom-ilm-policy]]
==== Create a custom ILM policy
Expand Down Expand Up @@ -197,6 +243,8 @@ and a delete phase after 30 days.
The `apm-transaction-metric-policy` keeps data in the hot, warm, and cold phases for a longer period of time,
and does not delete any data.

Additionally this example shows how to set custom rollover aliases.

[source,yml]
----
ilm:
Expand All @@ -205,12 +253,16 @@ and does not delete any data.
mapping:
- event_type: "error"
policy_name: "apm-error-span-policy"
rollover_alias: "apm-%{[observer.version]}-development-error"
- event_type: "span"
policy_name: "apm-error-span-policy"
rollover_alias: "apm-%{[observer.version]}-development-span"
- event_type: "transaction"
policy_name: "apm-transaction-metric-policy"
rollover_alias: "apm-%{[observer.version]}-development-transaction"
- event_type: "metric"
policy_name: "apm-transaction-metric-policy"
rollover_alias: "apm-%{[observer.version]}-development-metric"
enabled: true
policies:
- name: "apm-error-span-policy"
Expand Down
Loading

0 comments on commit 4c7f516

Please sign in to comment.