[DO NOT MERGE] Standalone Beats OTel collector PoC

[rdner]

This is the results of our experiment embedding OTel collector into Beats for running them in the standalone mode.

--

Steps to run filebeat in otel mode:

1. Clone from this branch.
2. Build the filebeat binary with mage package
3. Run the binary using

./filebeat otel --config /path/to/otel/config

The filebeat-otel mode makes use of filebeatreceiver and config of that same as that of standalone filebeat.

---

Currently, we are using hardcoded OpenTelemetry configurations, as the conversion from Filebeat to OpenTelemetry is nearly complete. There are still a few remaining tasks to address, and we’re actively working on them.

[VihasMakwana]

Attaching a demo of docker autodiscovery using filebeat (standalone otel mode)

Untitled_Project_V1.mp4

cc: @cmacknz

[belimawr]

Some key points regarding converting the Elasticsearch output configuration to the Elasticsearch OTel exporter:

1. Beats uses a custom TLS implementation, so currently only none and strict are supported
2. ILM management is done by Beats during startup, it's not directly tied to the output
3. None of the templateing features for index and pipelines are supported
4. Defining the pipeline or index on the input has not been tested

Testing the output config conversion

There is a test on libbeat/outputs/elasticsearch/config_otel_test.go that shows how the conversion can be used, libbeat/outputs/elasticsearch/testdata/filebeat.yml is an example configuration covering most of the currently supported fields.

[AndersonQ]

Kafka config converter key points:

• supporters config converted
• topic is supported ONLY if it's a constant or just the value of an attribute
• TLS: TODO - use the same as ES output
• not tested with OTel kafka exporter
• need more tests and review the current ones

[belimawr]

Folks, I've been trying to test it but I could not get my ES config translation function to work/be called, so I made some changes, now it is used, however as it is it's first test, there are some issues I haven't fixed yet.

Currently I'm using a hack to get the ES output configuration and use it as OTel ES exporter:

1. In cmd/otel.yml I added an output section with the Beats ES output configuration, that's the one parsed by

   beats/x-pack/filebeat/cmd/customProvider/beatprovider.go

   Line 43 in 216cf68

   esCfg, err := elasticsearch.ToOTelConfig(cfg)

2. For some reason skipping TLS validation is not working (one of the many things to fix), so I'm using a Cloud deployment.

The way I'm testing/running:

cd x-pack/filebeat
go build .
./filebeat otel -e -v -d "*" -config filebeat:./cmd/otel.yml

I can see the config is correctly parsed, converted and used to create an Elasticsearch exporter, however the index seems to be missing. If you let it running for a while you'll get an error from the exporter.

[belimawr]

I finally managed to get all the different bits working together 🎉

TL;DR:
Disabling TLS is not working, it might be an issue in the Elasticsearch exporter, so just make sure to either not use TLS or use an ES/Kibana with valid certificates. If you want the dataview to be automatically created, you need to setup the Kibana host and credentials.
The configuration file for the demo is x-pack/filebeat/cmd/filebeat.yml

cd x-pack/filebeat
# This will setup ILM, Index templates, data view, etc
go run . setup -e -v
# Actually run Filebeat as OTel collector using the converted ES output settings
go run . otel -e -v -d "*" -config filebeat:./cmd/filebeat.yml

Known issues:

• Disabling TLS validation is not working as expected.
• cloud.id cannot be used to run Filebeat because it clashes with the output.otelconsumer we manually insert.
• The only supported output is Elasticsearch because assemble an OTel config based on the filebeat.yml passed by picking some pieces like the output settings.

[rdner]

This PoC got polished and finalized in #41849
Closing.