Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add ability to run components in the Otel manager #6697

Merged
merged 15 commits into from
Feb 24, 2025
Merged

Add ability to run components in the Otel manager #6697

merged 15 commits into from
Feb 24, 2025

Conversation

swiatekm
Copy link
Contributor

@swiatekm swiatekm commented Feb 3, 2025 •
edited by jlind23
Loading

What does this PR do?

Allows to run components in the Otel manager instead of the runtime manager. Currently, only filestream is supported. This implements option 2 from https://github.com/elastic/ingest-dev/issues/4931.

Some notes about the implementation:

  • We use the same otel collector as hybrid mode. We add a prefix to each component and pipeline name in an effort to avoid conflicts.
  • The Otel configuration is generated from the component model agent already uses for beats processes. It is easier to manipulate and lets us emit the same status information in the future.
  • This behaviour is controlled by a single attribute of the Coordinator, the only way to enable it is by changing the code. This is a temporary measure until we decide what feature flags we want for it.
  • Status reporting uses the otel collector status, and doesn't try to be compatible with beats processes status.
  • Which input types are supported is hardcoded.

Why is it important?

This is a preprequisite for migrating to Otel within agent, which is our long-term plan. In the short-term, it will make it easier for us to run tests.

Checklist

  • I have read and understood the pull request guidelines of this project.
  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an integration test or an E2E test

How to test this PR locally

  1. Change the boolean value in

    elastic-agent/internal/pkg/agent/application/coordinator/coordinator.go

    Line 402 in 0c79718

    runComponentsInOtelManager: false, // change this to run supported components in the Otel manager
  2. Build an agent package
  3. Run agent with a configuration containing a filestream input

Related issues

@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 3, 2025

This pull request does not have a backport label. Could you fix it @swiatekm? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

@swiatekm swiatekm force-pushed the feat/otel-input-translation branch 2 times, most recently from ded1630 to 8f1baf5 Compare February 3, 2025 12:09
@swiatekm swiatekm force-pushed the feat/otel-input-translation branch 2 times, most recently from 3900e92 to 0c79718 Compare February 3, 2025 13:56
@swiatekm swiatekm mentioned this pull request Feb 3, 2025
Closed
13 tasks
@swiatekm swiatekm force-pushed the feat/otel-input-translation branch 6 times, most recently from 17861ca to 652cc95 Compare February 5, 2025 16:08
@mauri870 mauri870 mentioned this pull request Feb 5, 2025
Merged
8 tasks
@swiatekm swiatekm force-pushed the feat/otel-input-translation branch from 652cc95 to 85d2c2f Compare February 7, 2025 14:31
@swiatekm swiatekm force-pushed the feat/otel-input-translation branch 2 times, most recently from 22485fa to 9aa5de4 Compare February 10, 2025 11:40
@swiatekm swiatekm mentioned this pull request Feb 10, 2025
Merged
2 tasks
This was referenced Feb 10, 2025
Merged
Merged
@swiatekm swiatekm mentioned this pull request Feb 10, 2025
Merged
4 tasks
This was referenced Feb 11, 2025
Merged
Merged
@swiatekm swiatekm force-pushed the feat/otel-input-translation branch 3 times, most recently from c0ec532 to f453806 Compare February 11, 2025 16:32
@swiatekm swiatekm force-pushed the feat/otel-input-translation branch 2 times, most recently from 27138a9 to 17ee4dc Compare February 21, 2025 09:52
@swiatekm swiatekm force-pushed the feat/otel-input-translation branch from 17ee4dc to 70a222b Compare February 21, 2025 11:42
@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Issues
4 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
82.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@swiatekm swiatekm merged commit eeffc0e into main Feb 24, 2025
14 checks passed
@swiatekm swiatekm deleted the feat/otel-input-translation branch February 24, 2025 15:51
mergify bot pushed a commit that referenced this pull request Feb 24, 2025
@swiatekm @mergify
Add ability to run components in the Otel manager (#6697)
fc56291 
* Add ability to run components in the Otel manager

# Conflicts:
#	NOTICE.txt
#	go.mod
#	go.sum

* Add coordinator test

* Set metricbeat receiver signal type to logs

* Drop unnecessary transform processor

The conversion now happens in the otel consumer in beats.

* Determine default datastream type from beat name

* Fix diagnostics tests

* Promote output queue settings to receivers

* Move otel config translation to the otel package

* Emit the otel component diagnostic conditionally

* Add more otel config translation tests

* Code review fixes

* Fix diagnostics tests

* Code Review fixes

* Correctly set input types if not present

* More code review fixes

(cherry picked from commit eeffc0e)
@mergify mergify bot mentioned this pull request Feb 24, 2025
Merged
5 tasks
swiatekm added a commit that referenced this pull request Feb 25, 2025
@swiatekm
Add ability to run components in the Otel manager (#6697)
b8f1463 
* Add ability to run components in the Otel manager

# Conflicts:
#	NOTICE.txt
#	go.mod
#	go.sum

* Add coordinator test

* Set metricbeat receiver signal type to logs

* Drop unnecessary transform processor

The conversion now happens in the otel consumer in beats.

* Determine default datastream type from beat name

* Fix diagnostics tests

* Promote output queue settings to receivers

* Move otel config translation to the otel package

* Emit the otel component diagnostic conditionally

* Add more otel config translation tests

* Code review fixes

* Fix diagnostics tests

* Code Review fixes

* Correctly set input types if not present

* More code review fixes

(cherry picked from commit eeffc0e)
swiatekm added a commit that referenced this pull request Feb 25, 2025
@mergify @swiatekm
Add ability to run components in the Otel manager (#6697) (#6995)
f9eb06f 
* Add ability to run components in the Otel manager

# Conflicts:
#	NOTICE.txt
#	go.mod
#	go.sum

* Add coordinator test

* Set metricbeat receiver signal type to logs

* Drop unnecessary transform processor

The conversion now happens in the otel consumer in beats.

* Determine default datastream type from beat name

* Fix diagnostics tests

* Promote output queue settings to receivers

* Move otel config translation to the otel package

* Emit the otel component diagnostic conditionally

* Add more otel config translation tests

* Code review fixes

* Fix diagnostics tests

* Code Review fixes

* Correctly set input types if not present

* More code review fixes

(cherry picked from commit eeffc0e)

Co-authored-by: Mikołaj Świątek <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cmacknz cmacknz cmacknz approved these changes

@blakerouse blakerouse blakerouse approved these changes

@mauri870 mauri870 mauri870 approved these changes

@leehinman leehinman leehinman approved these changes

@khushijain21 khushijain21 khushijain21 approved these changes

@michalpristas michalpristas Awaiting requested review from michalpristas michalpristas is a code owner automatically assigned from elastic/elastic-agent-control-plane

@pchila pchila Awaiting requested review from pchila

Assignees

@swiatekm swiatekm

Labels
backport-8.x Automated backport to the 8.x branch with mergify backport-skip skip-changelog Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@swiatekm @cmacknz @elasticmachine @pkoutsovasilis @blakerouse @mauri870 @leehinman @khushijain21 @pchila @pierrehilbert