Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add missing kubernetes fields; update istio doc to use kubernetes.pod.ip instead of host #37578

Merged
merged 6 commits into from
Jan 22, 2024
Merged

Add missing kubernetes fields; update istio doc to use kubernetes.pod.ip instead of host #37578

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a93c80d
add missing kubernetes fields; update istio doc to use kubernetes.pod…
tetianakravchenko Jan 8, 2024
00cb450
extend documentation with full list of fields; fix add_resource_metad…
tetianakravchenko Jan 10, 2024
2fbf545
test all variables and adjust documentation accordingly
tetianakravchenko Jan 15, 2024
576106a
Merge branch 'main' into autodiscover-documentation
tetianakravchenko Jan 15, 2024
6041d3f
Update libbeat/docs/shared-autodiscover.asciidoc
tetianakravchenko Jan 18, 2024
e9371aa
Update libbeat/docs/shared-autodiscover.asciidoc
tetianakravchenko Jan 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions libbeat/docs/shared-autodiscover.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,10 +203,10 @@ Example:
Different Beats that refer to the same leader lease will be competitors in holding the lease
and only one will be elected as leader each time.

The configuration of templates and conditions is similar to that of the Docker provider. Configuration templates can
contain variables from the autodiscover event. They can be accessed under data namespace.
Configuration templates can contain variables from the autodiscover event. These variables can be accessed under the `data`
namespace, e.g. to access Pod IP: `${data.kubernetes.pod.ip}`.

These are the fields available within config templating. The `kubernetes.*` fields will be available on each emitted event.
These are the fields available within config templating. The `kubernetes.*` fields will be available on each emitted event:

[float]
====== Generic fields:
Expand All @@ -221,18 +221,22 @@ These are the fields available within config templating. The `kubernetes.*` fiel
* kubernetes.container.image
* kubernetes.container.name
* kubernetes.namespace
* kubernetes.namespace_uid
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* kubernetes.namespace_uid
* kubernetes.namespace.uid

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The field name is indeed kubernetes.namespace_uid.
@tetianakravchenko maybe we should find a way in this document to add also kubernetes.labels, kubernetes.annotations, kubernetes.namespace_labels and kubernetes.namespace_annotations. Those metadata fields are also part of the metadata fields that can be used in templating when the right parameters are set.
There is this section in the doc that states

If the `include_annotations` config is added to the provider config, then the list of annotations present in the config
are added to the event.

which is ok as an information but we don't list the field. Maybe we could list all fields above and leave those comments there as is.
Also we could add some comments about the namespace_labels and namespace_annotations based on the add_resource_metadata configuration.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gsantoro there is no such field as kubernetes.namespace.uid :

Screenshot 2024-01-10 at 14 18 30

I see that we have such field documented for elastic-agent - https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html, I believe it is a mistake in the elastic-agent documentation

Copy link
Contributor

@gsantoro gsantoro Jan 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MichaelKatsoulis I am trying to wrap my head around the docs and the implementation.

I can found two references kubernetes dynamic variables at https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html and https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover.html. For example kubernetes.namespace.uid is in the first one and not in the second doc.

Should those two docs have the same variables and both of them have kubernetes.namespace_uid?

what about this code from elastic-agent-autodiscover where the docs point at namespace.uid and the test at namespace_uid? at https://github.com/elastic/elastic-agent-autodiscover/blob/4f4898db82e5a5b37eab7e0e093435b9fb739461/kubernetes/metadata/namespace_test.go#L69

I could use some clarification about where in the code those variables are created.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gsantoro kubernetes.namespace is unfortunately a keyword and not an object. There was a try in 8.0 to do this breaking change and put all namespace related fields under the kubernetes.namespace object but it was reverted before it reached production. This was done mainly because of the possible problems it would create to existing customers and the not easy migration procedure.

The documentation in https://www.elastic.co/guide/en/fleet/current/kubernetes-provider.html is wrong. The part of the code where those values are added is here https://github.com/elastic/elastic-agent-autodiscover/blob/b42be0df903aba6580b80761af6df86fa80c845d/kubernetes/metadata/namespace.go#L117

* kubernetes.node.name
* kubernetes.pod.name
* kubernetes.pod.uid
* kubernetes.pod.ip

[float]
====== Node specific:
* kubernetes.node.name
* kubernetes.node.uid
* kubernetes.node.labels

[float]
====== Service specific:
* kubernetes.namespace
* kubernetes.namespace_uid
* kubernetes.service.name
* kubernetes.service.uid
* kubernetes.annotations
Expand Down
2 changes: 1 addition & 1 deletion x-pack/metricbeat/module/istio/proxy/_meta/docs.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,5 +29,5 @@ metricbeat.autodiscover:
config:
- module: istio
metricsets: ["proxy"]
hosts: "${data.host}:15090"
hosts: "${data.kubernetes.pod.ip}:15090"
--------------------------------------------
Loading