[otel/kube-stack]: add gateway collector configuration (#6444)

* feat: move telemetry aggregation and forwarding to gateway

* ci: use Elastic envs in gateway

* chore: add changelog entry

* fix: format values file

* feat: add apm loadbalancing

* chore: increase resource limits

* revert resource limits increase

* chore: remove config warnings

* docs: add Gateway collectors section

* revert: enable daemonset storagechecks

* rename metrics/otel pipeline and use signaltometrics

* unify k8s and host metrics pipelines

* use default traceID as loadbalancing routing_key

* chore: reuse k8s integration test helpers

* format values with Helm linter

* replace loadbalancing in favor of headless otlp

* Update testing/integration/otel_helm_test.go

Co-authored-by: Panos Koutsovasilis <[email protected]>

* Update testing/integration/otel_helm_test.go

Co-authored-by: Panos Koutsovasilis <[email protected]>

* rename k8s values options helper function

* move process attributes remove processor to gateway

* add batch processor for aggregation pipeline

* enable compression for cluster otlp connections

* chore: remove elastic endpoint references

* fix: do not generate service's signals for non apm data

* Revert "fix: do not generate service's signals for non apm data"

This reverts commit ffa6620.

* fix: set agent.name as edot-collector

* fix: enable daemon hostNetwork

* set unknown as default signaltometrics agent.name resource attribute

* remove signaltometrics for metrics-only services

---------

Co-authored-by: Panos Koutsovasilis <[email protected]>
(cherry picked from commit daed81e)

changelog/fragments/1735306293-add-gateway-collector-in-Helm-kube-stack-deployment.yaml

@@ -0,0 +1,32 @@
+# Kind can be one of:
+# - breaking-change: a change to previously-documented behavior
+# - deprecation: functionality that is being removed in a later release
+# - bug-fix: fixes a problem in a previous version
+# - enhancement: extends functionality but does not break or fix existing behavior
+# - feature: new functionality
+# - known-issue: problems that we are aware of in a given version
+# - security: impacts on the security of a product or a user’s deployment.
+# - upgrade: important information for someone upgrading from a prior version
+# - other: does not fit into any of the other categories
+kind: feature
+
+# Change summary; a 80ish characters long description of the change.
+summary: add gateway collector in Helm kube-stack deployment
+
+# Long description; in case the summary is not enough to describe the change
+# this field accommodate a description without length limits.
+# NOTE: This field will be rendered only for breaking-change and known-issue kinds at the moment.
+#description:
+
+# Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc.
+component: "elastic-agent"
+
+# PR URL; optional; the PR number that added the changeset.
+# If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added.
+# NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number.
+# Please provide it if you are adding a fragment for a different PR.
+#pr: https://github.com/owner/repo/1234
+
+# Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of).
+# If not present is automatically filled by the tooling with the issue linked to the PR number.
+#issue: https://github.com/owner/repo/1234

deploy/helm/edot-collector/kube-stack/README.md

@@ -17,15 +17,26 @@ The DaemonSet collectors handle the following data:
 - Logs: Utilizes [File Log Receiver receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver#readme) to gather logs from all Pods running on the respective node.
 - OTLP Traces: Utilizes [OTLP Receiver]( https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/otlpreceiver#readme) which configures both HTTP and GRPC endpoints on the node to receive OTLP trace data.
 
-### Deployment collector
+### Deployment collectors
+
+#### Cluster
 
 The OpenTelemetry components deployed within a Deployment collector focus on gathering data at the cluster level rather than at individual nodes.  A Deployment instance of the collector operates as a standalone (unlike DaemonSet collector instances, which are deployed on every node)
 
-The Deployment collector handles the following data:
+The Cluster Deployment collector handles the following data:
 
 - Kubernetes Events: Monitors and collects events occurring across the entire Kubernetes cluster, utilizing [Kubernetes Objects Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sobjectsreceiver#readme).
 - Cluster Metrics: Captures metrics that provide insights into the overall health and performance of the Kubernetes cluster, utilizing [Kubernetes Cluster Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver#readme).
 
+#### Gateway
+
+The OpenTelemetry components deployed within the `Gateway` Deployment collectors focus on processing and exporting OTLP data to Elasticsearch. Processing components:
+
+- [Elastic Trace processor](https://github.com/elastic/opentelemetry-collector-components/tree/main/processor/elastictraceprocessor): The processor enriches traces with elastic specific requirements. It uses opentelemetry-lib to perform the actual enrichments.
+- [Elastic Infra Metrics processor](https://github.com/elastic/opentelemetry-collector-components/tree/main/processor/elasticinframetricsprocessor): The Elastic Infra Metrics Processor is used to bridge the gap between OTEL and Elastic Infra Metrics.
+- [LSM interval processor](https://github.com/elastic/opentelemetry-collector-components/tree/main/processor/lsmintervalprocessor): [Interval processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/intervalprocessor) with db-backed persistence.
+- [Signal to Metrics connector](https://github.com/elastic/opentelemetry-collector-components/tree/main/connector/signaltometricsconnector): Produces metrics from all signal types (traces, logs, or metrics).
+
 ### Auto-instrumentation
 
 The Helm Chart is configured to enable zero-code instrumentation using the [Operator's Instrumentation resource](https://github.com/open-telemetry/opentelemetry-operator/?tab=readme-ov-file#opentelemetry-auto-instrumentation-injection) for the following programming languages: