From 3ea8affc117c77b7fb8cfd0537eea5eea7ade9f6 Mon Sep 17 00:00:00 2001 From: Fae Charlton Date: Wed, 6 Dec 2023 11:03:56 -0500 Subject: [PATCH 1/4] Document the Elasticsearch output's 'preset' field --- .../elasticsearch/docs/elasticsearch.asciidoc | 55 ++++++++++++++++++- 1 file changed, 54 insertions(+), 1 deletion(-) diff --git a/libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc b/libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc index 1b84948b2779..0c2345a9183d 100644 --- a/libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc +++ b/libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc @@ -701,7 +701,7 @@ The default is 3s. The http request timeout in seconds for the Elasticsearch request. The default is 90. -==== `allow_older_versions` +===== `allow_older_versions` By default, {beatname_uc} expects the Elasticsearch instance to be on the same or newer version to provide optimal experience. We suggest you connect to the same version to make sure all features {beatname_uc} is using are @@ -759,6 +759,59 @@ output.elasticsearch: index: "my-dead-letter-index" ------------------------------------------------------------------------------ +===== `preset` + +The performance preset to apply to the output configuration. Performance presets apply a set of configuration overrides based on a desired performance goal. If set, a performance preset will override other configuration flags to match the recommended settings for that preset. Valid options are `balanced` (good starting point for general efficiency), `throughput` (good for high data volumes, may increase cpu and memory requirements), `scale` (reduces ambient resource use in large low-throughput deployments), `latency` (minimize the time for fresh data to become visible in Elasticsearch), and `custom` (apply user configuration directly with no overrides). The default if unspecified is `custom`. + +Presets represent current recommendations based on the intended goal; their internal effect may change between versions. Currently the presets have the following effects: + +[cols="2,1,1,1,1"] +|=== +|preset |balanced |throughput |scale |latency + +|`bulk_max_size` +|1600 +|1600 +|1600 +|50 + +|`worker` +|1 +|4 +|1 +|1 + +|`queue.mem.events` +|3200 +|12800 +|3200 +|4100 + +|`queue.mem.flush.min_events` +|1600 +|1600 +|1600 +|2050 + +|`queue.mem.flush.timeout` +|`10s` +|`5s` +|`20s` +|`1s` + +|`compression_level` +|1 +|1 +|1 +|1 + +|`idle_connection_timeout` +|`3s` +|`15s` +|`1s` +|`60s` +|=== + [[es-apis]] ==== Elasticsearch APIs {beatname_uc} will use the `_bulk` API from {es}, the events are sent From a8d4f04ad64073650c6d8da12e8f1ff8fc00fccc Mon Sep 17 00:00:00 2001 From: Fae Charlton Date: Thu, 7 Dec 2023 10:27:40 -0500 Subject: [PATCH 2/4] add example of preset configuration --- .../outputs/elasticsearch/docs/elasticsearch.asciidoc | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc b/libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc index 0c2345a9183d..92bd259870e6 100644 --- a/libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc +++ b/libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc @@ -761,7 +761,16 @@ output.elasticsearch: ===== `preset` -The performance preset to apply to the output configuration. Performance presets apply a set of configuration overrides based on a desired performance goal. If set, a performance preset will override other configuration flags to match the recommended settings for that preset. Valid options are `balanced` (good starting point for general efficiency), `throughput` (good for high data volumes, may increase cpu and memory requirements), `scale` (reduces ambient resource use in large low-throughput deployments), `latency` (minimize the time for fresh data to become visible in Elasticsearch), and `custom` (apply user configuration directly with no overrides). The default if unspecified is `custom`. +The performance preset to apply to the output configuration. + +["source","yaml"] +------------------------------------------------------------------------------ +output.elasticsearch: + hosts: ["http://localhost:9200"] + preset: balanced +------------------------------------------------------------------------------ + +Performance presets apply a set of configuration overrides based on a desired performance goal. If set, a performance preset will override other configuration flags to match the recommended settings for that preset. Valid options are `balanced` (good starting point for general efficiency), `throughput` (good for high data volumes, may increase cpu and memory requirements), `scale` (reduces ambient resource use in large low-throughput deployments), `latency` (minimize the time for fresh data to become visible in Elasticsearch), and `custom` (apply user configuration directly with no overrides). The default if unspecified is `custom`. Presets represent current recommendations based on the intended goal; their internal effect may change between versions. Currently the presets have the following effects: From fb15ecdfe2c433c2f88be8b80ecd273effe959e7 Mon Sep 17 00:00:00 2001 From: Fae Charlton Date: Mon, 8 Jan 2024 17:45:45 -0500 Subject: [PATCH 3/4] Review comments --- libbeat/docs/queueconfig.asciidoc | 3 ++ .../elasticsearch/docs/elasticsearch.asciidoc | 29 +++++++++++++------ 2 files changed, 23 insertions(+), 9 deletions(-) diff --git a/libbeat/docs/queueconfig.asciidoc b/libbeat/docs/queueconfig.asciidoc index f4e2d62c6eae..08ece0f752f5 100644 --- a/libbeat/docs/queueconfig.asciidoc +++ b/libbeat/docs/queueconfig.asciidoc @@ -61,6 +61,7 @@ queue.mem: You can specify the following options in the `queue.mem` section of the +{beatname_lc}.yml+ config file: [float] +[[queue-mem-events-option]] ===== `events` Number of events the queue can store. This value should be evenly divisible by `flush.min_events` to @@ -69,6 +70,7 @@ avoid sending partial batches to the output. The default value is 3200 events. [float] +[[queue-mem-flush-min-events-option]] ===== `flush.min_events` Minimum number of events required for publishing. If this value is set to 0 or 1, events are @@ -80,6 +82,7 @@ sent by the output. The default value is 1600. [float] +[[queue-mem-flush-timeout-option]] ===== `flush.timeout` Maximum wait time for `flush.min_events` to be fulfilled. If set to 0s, events are available to the diff --git a/libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc b/libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc index 92bd259870e6..1e9ae2d661af 100644 --- a/libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc +++ b/libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc @@ -98,6 +98,7 @@ output.elasticsearch: In the previous example, the Elasticsearch nodes are available at `https://10.45.3.2:9220/elasticsearch` and `https://10.45.3.1:9230/elasticsearch`. +[[compression-level-option]] ===== `compression_level` The gzip compression level. Setting this value to `0` disables compression. @@ -114,6 +115,7 @@ Configure escaping of HTML in strings. Set to `true` to enable escaping. The default value is `false`. +[[worker-option]] ===== `worker` The number of workers per configured host publishing events to Elasticsearch. This @@ -659,6 +661,7 @@ The default is 3. endif::[] +[[bulk-max-size-option]] ===== `bulk_max_size` The maximum number of events to bulk in a single Elasticsearch bulk API index request. The default is 1600. @@ -691,6 +694,7 @@ default is `1s`. The maximum number of seconds to wait before attempting to connect to Elasticsearch after a network error. The default is `60s`. +[[idle-connection-timeout-option]] ===== `idle_connection_timeout` The maximum amount of time an idle connection will remain idle before closing itself. @@ -770,51 +774,58 @@ output.elasticsearch: preset: balanced ------------------------------------------------------------------------------ -Performance presets apply a set of configuration overrides based on a desired performance goal. If set, a performance preset will override other configuration flags to match the recommended settings for that preset. Valid options are `balanced` (good starting point for general efficiency), `throughput` (good for high data volumes, may increase cpu and memory requirements), `scale` (reduces ambient resource use in large low-throughput deployments), `latency` (minimize the time for fresh data to become visible in Elasticsearch), and `custom` (apply user configuration directly with no overrides). The default if unspecified is `custom`. +Performance presets apply a set of configuration overrides based on a desired performance goal. If set, a performance preset will override other configuration flags to match the recommended settings for that preset. Valid options are: +* `balanced` (good starting point for general efficiency) +* `throughput` (good for high data volumes, may increase cpu and memory requirements) +* `scale` (reduces ambient resource use in large low-throughput deployments) +* `latency` (minimize the time for fresh data to become visible in Elasticsearch) +* `custom` (apply user configuration directly with no overrides) -Presets represent current recommendations based on the intended goal; their internal effect may change between versions. Currently the presets have the following effects: +The default if unspecified is `custom`. + +Presets represent current recommendations based on the intended goal; their effect may change between versions to better suit those goals. Currently the presets have the following effects: [cols="2,1,1,1,1"] |=== |preset |balanced |throughput |scale |latency -|`bulk_max_size` +|<> |1600 |1600 |1600 |50 -|`worker` +|<> |1 |4 |1 |1 -|`queue.mem.events` +|<> |3200 |12800 |3200 |4100 -|`queue.mem.flush.min_events` +|<> |1600 |1600 |1600 |2050 -|`queue.mem.flush.timeout` +|<> |`10s` |`5s` |`20s` |`1s` -|`compression_level` +|<> |1 |1 |1 |1 -|`idle_connection_timeout` +|<> |`3s` |`15s` |`1s` From 35afddabc4387a6c2f3769e3a267c292f00df044 Mon Sep 17 00:00:00 2001 From: Fae Charlton Date: Mon, 8 Jan 2024 18:21:46 -0500 Subject: [PATCH 4/4] edits --- .../outputs/elasticsearch/docs/elasticsearch.asciidoc | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc b/libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc index 1e9ae2d661af..046c45a34dc1 100644 --- a/libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc +++ b/libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc @@ -775,11 +775,11 @@ output.elasticsearch: ------------------------------------------------------------------------------ Performance presets apply a set of configuration overrides based on a desired performance goal. If set, a performance preset will override other configuration flags to match the recommended settings for that preset. Valid options are: -* `balanced` (good starting point for general efficiency) -* `throughput` (good for high data volumes, may increase cpu and memory requirements) -* `scale` (reduces ambient resource use in large low-throughput deployments) -* `latency` (minimize the time for fresh data to become visible in Elasticsearch) -* `custom` (apply user configuration directly with no overrides) +* `balanced`: good starting point for general efficiency +* `throughput`: good for high data volumes, may increase cpu and memory requirements +* `scale`: reduces ambient resource use in large low-throughput deployments +* `latency`: minimize the time for fresh data to become visible in Elasticsearch +* `custom`: apply user configuration directly with no overrides The default if unspecified is `custom`.