[filebeat] Introduce otel mode

[khushijain21]

To note

• This PR uses converters for config translation. This will help reuse and maintain translation logic across beats.
• All common "beats to OTel" logic is under libbeat/otelbeat/
• Output specific config translation logic is under libbeat/outputs/${output_type}
• It throws error if any output other than elasticsearch is configured
• It throws error when unsupported elasticsearch paramaters are configured (some of them are just documented tho)

Elasticsearch paramaters that are currently unsupported

indices
pipelines
parameters
preset
setup.ilm.* -> supported but the logic is not in place yet
proxy_disable -> supported but the logic is not in place yet
proxy_headers
compression_level
loadbalance
non_indexable_policy
allow_older_versions
escape_html
kerberos-*
bulk_max_size

and some TLS paramaters that are currently not supported

// currently unsupported parameters
// ssl.curve_types
// ssl.ca_sha256
// ssl.ca_trustred_fingerprint
// ssl.renegotiation
// ssl.verification_mode:
// ssl.supported_protocols -> partially supported
// ssl.restart_on_cert_change.*
// ssl.client_authentication

The change in filebeat and agentbeat binary sizes before and after this changeset is as follows

// For filebeat before
➜  golang-crossbuild git:(main) ✗ du -h filebeat-darwin-arm64
141M    filebeat-darwin-arm64

➜  golang-crossbuild git:(fosb1) ✗ du -h filebeat-darwin-arm64
158M    filebeat-darwin-arm64

// In Agentbeat

➜  golang-crossbuild git:(main) ✗ du -h agentbeat-darwin-arm64
222M    agentbeat-darwin-arm64

➜  golang-crossbuild git:(main) ✗ du -h agentbeat-darwin-arm64
242M    agentbeat-darwin-arm64

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

cd x-pack/filebeat
mage build
./filebeat otel
# or 
./filebeat otel --config <filename>

Elasticsearch ouput was tested manually. The following are the steps

Run

cd filebeat
mage docker:composeUp

Add following config to filebeat-otel.yml in x-pack/filebeat/

filebeat.inputs:
  - type: filestream
    id: filestream-input-id
    enabled: true
    paths:
      - ./test.log

output:
  elasticsearch:
    hosts: ["http://localhost:9200"]
    username: admin
    password: testing

setup.kibana:
  host: http://localhost:5601
  username: admin
  password: testing

Add dummy text to test.log and you can verify they are reaching kibana on localhost:5601

It was also manually tested with elasticsearch service

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @khushijain21? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[VihasMakwana]

Overall looks good. Just need to address lint failures and handle errors.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fosb1 upstream/fosb1
git merge upstream/main
git push upstream fosb1

[khushijain21]

think the -c option might be getting ignored:

so we cannot use -c shorthand with otel command, as it is a non-unique global shorthand.

And it works correctly now with this cmd

./filebeat otel --config <filename>

[rdner]

Current state looks good to me.

I expect all the TODOs to be addressed in a follow up PR right after merging this.

[khushijain21]

/test

[jlind23]

@khushijain21 looks like you got enough approvals to move this over the line right?

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fosb1 upstream/fosb1
git merge upstream/main
git push upstream fosb1

[khushijain21]

/test