Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

otel: add test for document equivalence between agent filestream input and filebeat receiver #6681

Merged
merged 22 commits into from
Feb 14, 2025
Merged

otel: add test for document equivalence between agent filestream input and filebeat receiver #6681

merged 22 commits into from
Feb 14, 2025

Conversation

mauri870
Copy link
Member

@mauri870 mauri870 commented Jan 31, 2025 •
edited
Loading

What does this PR do?

This PR adds a test that compares a log line ingested by a filestream agent input and a filebeat OTel receiver with equivalent configuration.

Checklist

  • I have read and understood the pull request guidelines of this project.
  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool
  • I have added an integration test or an E2E test

Related issues

@mauri870 mauri870 added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team Cleanup skip-changelog backport-8.x Automated backport to the 8.x branch with mergify labels Jan 31, 2025
@mauri870 mauri870 self-assigned this Jan 31, 2025
@mauri870 mauri870 changed the title Hybrid agent test otel: add test for document equivalence between filebeat and filebeat receiver Jan 31, 2025
@mauri870 mauri870 changed the title otel: add test for document equivalence between filebeat and filebeat receiver otel: add test for document equivalence between agent filestream input and filebeat receiver Jan 31, 2025
@mauri870 mauri870 marked this pull request as ready for review January 31, 2025 14:03
@mauri870 mauri870 requested a review from a team as a code owner January 31, 2025 14:03
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@mauri870 mauri870 requested review from leehinman and removed request for kaanyalti January 31, 2025 14:03
leehinman
leehinman reviewed Jan 31, 2025
View reviewed changes
Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if you add the default processors listed at https://github.com/elastic/beats/blob/76f40863c9918ee1a691a2bb29cbab98a5d83878/x-pack/filebeat/cmd/root.go#L48 and add the processors listed in the generate.go below you should get a much shorter list of fields to ignore.

@mauri870 mauri870 marked this pull request as draft February 3, 2025 12:33
@mauri870 mauri870 marked this pull request as ready for review February 6, 2025 14:16
khushijain21
khushijain21 previously approved these changes Feb 7, 2025
View reviewed changes
@khushijain21 khushijain21 dismissed their stale review February 7, 2025 06:43

tests are failing

@pierrehilbert
Copy link
Contributor

I don't think the failing tests are related to this PR:

=== RUN   TestDownloadFile
    utils_test.go:21: 
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-aws-1738851441003538166/elastic/elastic-agent/dev-tools/mage/downloads/utils_test.go:21
        	Error:      	Expected nil, but got: &fmt.wrapError{msg:"downloading file https://www.elastic.co/robots.txt: GET request failed with 504", err:(*errors.errorString)(0xc00085a970)}
        	Test:       	TestDownloadFile
--- FAIL: TestDownloadFile (20.57s)

Will update the branch to have them running again and hope the gateway failure was temporary

@mergify Mergify
Copy link
Contributor

mergify bot commented Feb 10, 2025

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b hybrid-agent-test upstream/hybrid-agent-test
git merge upstream/main
git push upstream hybrid-agent-test

leehinman
leehinman approved these changes Feb 10, 2025
View reviewed changes
Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

@mauri870 mauri870 merged commit 41882ac into elastic:main Feb 14, 2025
14 checks passed
mergify bot pushed a commit that referenced this pull request Feb 14, 2025
@mauri870 @mergify
otel: add test for document equivalence between agent filestream inpu…
c846821 
…t and filebeat receiver (#6681)

* otel: add test for agent hybrid

* add missing fingerprint config options to index small files

* ensure new lines are written to the input file

* disable compression level so we inspect the requests

* add queue timeout to normal beats and separate path.home for fbreceiver

* add path.home for normal filebeat

* fix agent not starting

* print elastic-agent output if test fails

* add note for metadata field

* fail if ignored field is not present in both maps

* add host metadata processor, remove 'host.*' from ignored fields

* Use global processor

* more strict rules for ignored fields to avoid false positives

* use add_fields to add missing fields, use a single index

* add add_cloud_metadata processor

* avoid checking cmd.Wait error since it is bogus

(cherry picked from commit 41882ac)
mergify bot pushed a commit that referenced this pull request Feb 14, 2025
@mauri870 @mergify
otel: add test for document equivalence between agent filestream inpu…
bb70746 
…t and filebeat receiver (#6681)

* otel: add test for agent hybrid

* add missing fingerprint config options to index small files

* ensure new lines are written to the input file

* disable compression level so we inspect the requests

* add queue timeout to normal beats and separate path.home for fbreceiver

* add path.home for normal filebeat

* fix agent not starting

* print elastic-agent output if test fails

* add note for metadata field

* fail if ignored field is not present in both maps

* add host metadata processor, remove 'host.*' from ignored fields

* Use global processor

* more strict rules for ignored fields to avoid false positives

* use add_fields to add missing fields, use a single index

* add add_cloud_metadata processor

* avoid checking cmd.Wait error since it is bogus

(cherry picked from commit 41882ac)
This was referenced Feb 14, 2025
Merged
Merged
mauri870 added a commit that referenced this pull request Feb 14, 2025
@mergify @mauri870
otel: add test for document equivalence between agent filestream inpu…
2161adb 
…t and filebeat receiver (#6681) (#6868)

* otel: add test for agent hybrid

* add missing fingerprint config options to index small files

* ensure new lines are written to the input file

* disable compression level so we inspect the requests

* add queue timeout to normal beats and separate path.home for fbreceiver

* add path.home for normal filebeat

* fix agent not starting

* print elastic-agent output if test fails

* add note for metadata field

* fail if ignored field is not present in both maps

* add host metadata processor, remove 'host.*' from ignored fields

* Use global processor

* more strict rules for ignored fields to avoid false positives

* use add_fields to add missing fields, use a single index

* add add_cloud_metadata processor

* avoid checking cmd.Wait error since it is bogus

(cherry picked from commit 41882ac)

Co-authored-by: Mauri de Souza Meneguzzo <[email protected]>
mauri870 added a commit that referenced this pull request Feb 19, 2025
@mergify @mauri870
otel: add test for document equivalence between agent filestream inpu…
003c923 
…t and filebeat receiver (#6681) (#6867)

* otel: add test for agent hybrid

* add missing fingerprint config options to index small files

* ensure new lines are written to the input file

* disable compression level so we inspect the requests

* add queue timeout to normal beats and separate path.home for fbreceiver

* add path.home for normal filebeat

* fix agent not starting

* print elastic-agent output if test fails

* add note for metadata field

* fail if ignored field is not present in both maps

* add host metadata processor, remove 'host.*' from ignored fields

* Use global processor

* more strict rules for ignored fields to avoid false positives

* use add_fields to add missing fields, use a single index

* add add_cloud_metadata processor

* avoid checking cmd.Wait error since it is bogus

(cherry picked from commit 41882ac)

Co-authored-by: Mauri de Souza Meneguzzo <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@leehinman leehinman leehinman approved these changes

@khushijain21 khushijain21 khushijain21 approved these changes

@blakerouse blakerouse Awaiting requested review from blakerouse blakerouse is a code owner automatically assigned from elastic/elastic-agent-control-plane

Assignees

@mauri870 mauri870

Labels
backport-8.x Automated backport to the 8.x branch with mergify backport-9.0 Automated backport to the 9.0 branch Cleanup skip-changelog Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@mauri870 @elasticmachine @pierrehilbert @leehinman @khushijain21