[9.0](backport #6630) [Managed OTLP] Add Host/K8s onboarding configurations

[mergify]

This PR adds the configuration files for the EDOT collector that will be provided during the Kibana's OpenTelemetry Host and K8s onboarding flow for MOTel.
The files are a copy of the current sample configurations but replacing how the data is forwarded, instead of using the Elasticsearch exporter + Elastic processors/connector for data transformations, it directly sends the gathered OTLP signals to an OTLP ingest endpoint. For the Helm values file, only the gateway collector configuration has been modified.

Usage (Kibana)

Kibana instructions will point to the MOTel configuration (managed_otlp dir) or the managed one, depending on if it's a serverless env or not:

$ kubectl create namespace opentelemetry-operator-system
$ kubectl create secret generic elastic-secret-otel --namespace opentelemetry-operator-system --from-literal=elastic_endpoint='<YOUR_INGEST_URL>' --from-literal=elastic_api_key='<YOUR_BASE64_APIKEY>'
$ helm install opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack --namespace opentelemetry-operator-system --values <URL_MOTEL_CONFIG_ELASTIC_AGENT>/values.yaml --version '0.3.9'

Tests

Added an integration test for the new Helm configuration in testing/integration/otel_helm_test.go. The ELASTIC_ENDPOINT and ELASTIC_APIKEY values used during the test do not point to any real environment (MOTel only available in serverless atm), but currently we are only validating that the K8s pods are being created (no error in the config).

Alternative solutions

Unify data collection configurations

Managed and MOTel sample configurations share the same data collection settings. To simplify maintenance, it's preferable to reference a single configuration file. For the Helm deployment, we could do the following:

1. A single shared configuration (values.yaml) for data collection (daemon and cluster collectors)
2. A configuration file that contains the gateway collector config for managed environments and another one for MOTel.
3. Data collection + data forwarding/processing configurations must be used:

// Elasticsearch
$ helm install ... --values <DAEMON_CLUSTER_CONFIG> --values <GATEWAY_CONFIG> --version '0.3.9'

// Managed Otlp
$ helm install ... --values <DAEMON_CLUSTER_CONFIG> --values <GATEWAY_MOTEL_CONFIG> --version '0.3.9'

The same strategy could be applied for the onhost configurations, move the processing, exporters and pipelines configuration in a separate files and launch the collector by merging two configs:

// Elasticsearch
STORAGE_DIR=/tmp ELASTIC_ENDPOINT="https://localhost:8200" elastic-agent otel --config platformlogs_hostmetrics.yml --config es_pipeline.yml

// Managed Otlp
STORAGE_DIR=/tmp ELASTIC_ENDPOINT="https://localhost:8200" elastic-agent otel --config platformlogs_hostmetrics.yml --config motel_pipeline.yml

This solution eases the configuration maintainability (e.g. bumping deps), but at the cost of usability (two related configuration files).

cc @mlunadia

---

This is an automatic backport of pull request #6630 done by Mergify.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mergify]

This pull request has not been merged yet. Could you please review and merge it @rogercoll? 🙏

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 9cfad9a

Failed CI Steps

• Unit tests - Windows 2016

History

• 💔 Build #17329 failed cb7021c

cc @rogercoll

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube