Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document the Elasticsearch output's 'preset' field #37315

Merged
merged 5 commits into from
Jan 10, 2024
Merged

Document the Elasticsearch output's 'preset' field #37315

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3ea8aff
Document the Elasticsearch output's 'preset' field
faec Dec 6, 2023
a8d4f04
add example of preset configuration
faec Dec 7, 2023
fb15ecd
Review comments
faec Jan 8, 2024
e49b5b2
Merge branch 'main' of github.com:elastic/beats into preset-docs
faec Jan 8, 2024
35afdda
edits
faec Jan 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions libbeat/docs/queueconfig.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ queue.mem:
You can specify the following options in the `queue.mem` section of the +{beatname_lc}.yml+ config file:

[float]
[[queue-mem-events-option]]
===== `events`

Number of events the queue can store. This value should be evenly divisible by `flush.min_events` to
Expand All @@ -69,6 +70,7 @@ avoid sending partial batches to the output.
The default value is 3200 events.

[float]
[[queue-mem-flush-min-events-option]]
===== `flush.min_events`

Minimum number of events required for publishing. If this value is set to 0 or 1, events are
Expand All @@ -80,6 +82,7 @@ sent by the output.
The default value is 1600.

[float]
[[queue-mem-flush-timeout-option]]
===== `flush.timeout`

Maximum wait time for `flush.min_events` to be fulfilled. If set to 0s, events are available to the
Expand Down
75 changes: 74 additions & 1 deletion libbeat/outputs/elasticsearch/docs/elasticsearch.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ output.elasticsearch:
In the previous example, the Elasticsearch nodes are available at `https://10.45.3.2:9220/elasticsearch` and
`https://10.45.3.1:9230/elasticsearch`.

[[compression-level-option]]
===== `compression_level`

The gzip compression level. Setting this value to `0` disables compression.
Expand All @@ -114,6 +115,7 @@ Configure escaping of HTML in strings. Set to `true` to enable escaping.
The default value is `false`.


[[worker-option]]
===== `worker`

The number of workers per configured host publishing events to Elasticsearch. This
Expand Down Expand Up @@ -659,6 +661,7 @@ The default is 3.
endif::[]


[[bulk-max-size-option]]
===== `bulk_max_size`

The maximum number of events to bulk in a single Elasticsearch bulk API index request. The default is 1600.
Expand Down Expand Up @@ -691,6 +694,7 @@ default is `1s`.
The maximum number of seconds to wait before attempting to connect to
Elasticsearch after a network error. The default is `60s`.

[[idle-connection-timeout-option]]
===== `idle_connection_timeout`

The maximum amount of time an idle connection will remain idle before closing itself.
Expand All @@ -701,7 +705,7 @@ The default is 3s.

The http request timeout in seconds for the Elasticsearch request. The default is 90.

==== `allow_older_versions`
===== `allow_older_versions`
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an unrelated typo I fixed as long as I'm in here -- this field was on a different header level than all other fields in this section.


By default, {beatname_uc} expects the Elasticsearch instance to be on the same or newer version to provide
optimal experience. We suggest you connect to the same version to make sure all features {beatname_uc} is using are
Expand Down Expand Up @@ -759,6 +763,75 @@ output.elasticsearch:
index: "my-dead-letter-index"
------------------------------------------------------------------------------

===== `preset`

The performance preset to apply to the output configuration.

["source","yaml"]
------------------------------------------------------------------------------
output.elasticsearch:
hosts: ["http://localhost:9200"]
preset: balanced
------------------------------------------------------------------------------

Performance presets apply a set of configuration overrides based on a desired performance goal. If set, a performance preset will override other configuration flags to match the recommended settings for that preset. Valid options are:
* `balanced`: good starting point for general efficiency
* `throughput`: good for high data volumes, may increase cpu and memory requirements
* `scale`: reduces ambient resource use in large low-throughput deployments
* `latency`: minimize the time for fresh data to become visible in Elasticsearch
* `custom`: apply user configuration directly with no overrides

The default if unspecified is `custom`.

Presets represent current recommendations based on the intended goal; their effect may change between versions to better suit those goals. Currently the presets have the following effects:

[cols="2,1,1,1,1"]
|===
|preset |balanced |throughput |scale |latency

|<<bulk-max-size-option,`bulk_max_size`>>
|1600
|1600
|1600
|50

|<<worker-option,`worker`>>
|1
|4
|1
|1

|<<queue-mem-events-option,`queue.mem.events`>>
|3200
|12800
|3200
|4100

|<<queue-mem-flush-min-events-option,`queue.mem.flush.min_events`>>
|1600
|1600
|1600
|2050

|<<queue-mem-flush-timeout-option,`queue.mem.flush.timeout`>>
|`10s`
|`5s`
|`20s`
|`1s`

|<<compression-level-option,`compression_level`>>
|1
|1
|1
|1

|<<idle-connection-timeout-option,`idle_connection_timeout`>>
|`3s`
|`15s`
|`1s`
|`60s`
|===

[[es-apis]]
==== Elasticsearch APIs
{beatname_uc} will use the `_bulk` API from {es}, the events are sent
Expand Down
Loading