{,x-pack}/packetbeat: add support for pipeline module uploads #37291

efd6 · 2023-12-05T03:15:56Z

Proposed commit message

See title.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

For Move non-ECS fields in Network Packet Capture datastream fields out of root namespace integrations#8185

Use cases

Screenshots

Logs

elasticmachine · 2023-12-05T04:05:59Z

💚 Build Succeeded

Expand to view the summary

Build stats

Duration: 50 min 33 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-12-05T07:06:52Z

💚 Build Succeeded

Expand to view the summary

Build stats

Start Time: 2023-12-05T06:15:12.596+0000
Duration: 52 min 31 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2361
Skipped	25
Total	2386

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-12-07T03:51:46Z

❕ Build Aborted

Expand to view the summary

Build stats

Duration: 7 min 39 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-12-07T04:37:59Z

💚 Build Succeeded

Expand to view the summary

Build stats

Start Time: 2023-12-07T03:46:48.626+0000
Duration: 51 min 3 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2361
Skipped	25
Total	2386

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-12-08T06:40:21Z

❕ Build Aborted

Expand to view the summary

Build stats

Duration: 29 min 47 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-12-08T07:28:15Z

💚 Build Succeeded

Expand to view the summary

Build stats

Start Time: 2023-12-08T06:33:12.630+0000
Duration: 55 min 55 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2361
Skipped	25
Total	2386

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2023-12-08T08:14:04Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2023-12-08T08:17:54Z

💔 Build Failed

Expand to view the summary

Build stats

Start Time: 2023-12-08T08:12:31.028+0000
Duration: 5 min 0 sec

Steps errors

Expand to view the steps failures

`Error signal`

Took 0 min 0 sec . View more details here
Description: githubApiCall: The REST API call https://api.github.com/repos/elastic/beats/pulls/37291 return the message : java.lang.Exception: httpRequest: Failure connecting to the service https://api.github.com/repos/elastic/beats/pulls/37291 : httpRequest: Failure connecting to the service https://api.github.com/repos/elastic/beats/pulls/37291 : Code: 504Error: {"message": "We couldn't respond to your request in time. Sorry about that. Please try resubmitting your request and contact us if the problem persists."}

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

efd6 · 2023-12-08T20:07:01Z

/test

elasticmachine · 2023-12-08T20:58:42Z

💚 Build Succeeded

Expand to view the summary

Build stats

Start Time: 2023-12-08T20:07:17.222+0000
Duration: 51 min 18 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2361
Skipped	25
Total	2386

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2024-01-17T18:08:18Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b packetbeat_modules upstream/packetbeat_modules
git merge upstream/main
git push upstream packetbeat_modules

elasticmachine · 2024-01-17T20:51:35Z

💚 Build Succeeded

Expand to view the summary

Build stats

Duration: 54 min 54 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

andrewkroh

This looks good to me, but I think we need some accompanying documentation additions to explain the purpose of the pipelines and why you would want to ensure they are loaded. We'll need some docs to explain how to load the pipelines manually for cases where a non-Elasticsearch output is used. And lastly the docs will need to reflect the additional privilege that is needed to install the pipelines (manage_pipeline). This can be a separate PR.

andrewkroh · 2024-01-24T00:18:05Z

CHANGELOG.next.asciidoc

@@ -194,6 +194,8 @@ Setting environmental variable ELASTIC_NETINFO:false in Elastic Agent pod will d

 *Packetbeat*

+- Add metrics for TCP flags. {issue}36992[36992] {pull}36975[36975]


Is this changelog line intentional?

No it is not. Removed.

CHANGELOG.next.asciidoc

elasticmachine · 2024-01-24T02:33:06Z

💔 Build Failed

Expand to view the summary

Build stats

Duration: 49 min 49 sec

Pipeline error

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-25T04:07:25Z

💚 Build Succeeded

Expand to view the summary

Build stats

Duration: 49 min 21 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2024-01-25T16:17:12Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b packetbeat_modules upstream/packetbeat_modules
git merge upstream/main
git push upstream packetbeat_modules

[git-generate] yoink -pkg ./winlogbeat/module -dir packetbeat -y UploadPipelines,PipelinesFS gsed -r -i 's/Winlogbeat/Packetbeat/g' packetbeat/module/*.go gsed -r -i 's/winlogbeat/packetbeat/g' packetbeat/module/*.go goimports -w -local github.com/elastic ./packetbeat/module yoink -pkg ./x-pack/winlogbeat/module -dir x-pack/packetbeat -y init gsed -r -i 's/winlogbeat/packetbeat/g' x-pack/packetbeat/module/*.go mkdir -p x-pack/packetbeat/module/null/ingest touch x-pack/packetbeat/module/null/ingest/pin.yml

…ve placeholder [git-generate] data_stream_root=$(go env GOPATH)/src/github.com/elastic/integrations/packages/network_traffic/data_stream parallel mkdir -p x-pack/packetbeat/module/{/}/ingest ::: $data_stream_root/* parallel 'cp {}/elasticsearch/ingest_pipeline/* x-pack/packetbeat/module/{/}/ingest/' ::: $data_stream_root/* rm -rf x-pack/packetbeat/module/null

elasticmachine · 2024-01-25T21:25:05Z

💚 Build Succeeded

Expand to view the summary

Build stats

Start Time: 2024-01-25T20:35:38.380+0000
Duration: 49 min 20 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2361
Skipped	25
Total	2386

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

…c#37291) * packetbeat/module: new package for supporting modules [git-generate] yoink -pkg ./winlogbeat/module -dir packetbeat -y UploadPipelines,PipelinesFS gsed -r -i 's/Winlogbeat/Packetbeat/g' packetbeat/module/*.go gsed -r -i 's/winlogbeat/packetbeat/g' packetbeat/module/*.go goimports -w -local github.com/elastic ./packetbeat/module yoink -pkg ./x-pack/winlogbeat/module -dir x-pack/packetbeat -y init gsed -r -i 's/winlogbeat/packetbeat/g' x-pack/packetbeat/module/*.go mkdir -p x-pack/packetbeat/module/null/ingest touch x-pack/packetbeat/module/null/ingest/pin.yml * packetbeat/module: adjust pipeline names * packetbeat/beater: add support for pipeline uploads * x-pack/packetbeat/module: import pipelines from integrations and remove placeholder [git-generate] data_stream_root=$(go env GOPATH)/src/github.com/elastic/integrations/packages/network_traffic/data_stream parallel mkdir -p x-pack/packetbeat/module/{/}/ingest ::: $data_stream_root/* parallel 'cp {}/elasticsearch/ingest_pipeline/* x-pack/packetbeat/module/{/}/ingest/' ::: $data_stream_root/* rm -rf x-pack/packetbeat/module/null * x-pack/packetbeat/module: add routing

efd6 added enhancement Packetbeat Team:Security-External Integrations backport-skip Skip notification from the automated backport with mergify 8.12-candidate labels Dec 5, 2023

efd6 self-assigned this Dec 5, 2023

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Dec 5, 2023

efd6 mentioned this pull request Dec 7, 2023

Move non-ECS fields in Network Packet Capture datastream fields out of root namespace elastic/integrations#8185

Open

6 tasks

efd6 force-pushed the packetbeat_modules branch from d12b9c6 to 644f9f8 Compare December 7, 2023 03:44

efd6 force-pushed the packetbeat_modules branch 4 times, most recently from a1d759d to 561693e Compare December 8, 2023 06:32

efd6 marked this pull request as ready for review December 8, 2023 08:14

efd6 requested a review from a team as a code owner December 8, 2023 08:14

efd6 force-pushed the packetbeat_modules branch from 41cf122 to 8935221 Compare January 17, 2024 19:55

andrewkroh reviewed Jan 24, 2024

View reviewed changes

efd6 commented Jan 24, 2024

View reviewed changes

CHANGELOG.next.asciidoc Show resolved Hide resolved

efd6 force-pushed the packetbeat_modules branch from b6282ed to 08a161e Compare January 25, 2024 03:17

efd6 added 6 commits January 26, 2024 07:05

packetbeat/module: adjust pipeline names

8dfdf85

packetbeat/beater: add support for pipeline uploads

8518777

x-pack/packetbeat/module: add routing

b393f5e

add changelog entry

a68497d

efd6 force-pushed the packetbeat_modules branch from 08a161e to a68497d Compare January 25, 2024 20:35

efd6 requested a review from a team as a code owner January 25, 2024 20:35

andrewkroh approved these changes Jan 30, 2024

View reviewed changes

efd6 merged commit e900baf into elastic:main Jan 31, 2024
37 checks passed

efd6 mentioned this pull request Jan 31, 2024

packetbeat/docs: add documentation for ingest pipelines #37798

Merged

6 tasks

		@@ -194,6 +194,8 @@ Setting environmental variable ELASTIC_NETINFO:false in Elastic Agent pod will d

		Packetbeat

		- Add metrics for TCP flags. {issue}36992[36992] {pull}36975[36975]

{,x-pack}/packetbeat: add support for pipeline module uploads #37291

{,x-pack}/packetbeat: add support for pipeline module uploads #37291

Conversation

efd6 commented Dec 5, 2023 • edited Loading

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Dec 5, 2023

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

elasticmachine commented Dec 5, 2023

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Dec 7, 2023

❕ Build Aborted

Build stats

🤖 GitHub comments

elasticmachine commented Dec 7, 2023

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Dec 8, 2023

❕ Build Aborted

Build stats

🤖 GitHub comments

elasticmachine commented Dec 8, 2023

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Dec 8, 2023

elasticmachine commented Dec 8, 2023

💔 Build Failed

Build stats

Steps errors

Error signal

❕ Flaky test report

🤖 GitHub comments

efd6 commented Dec 8, 2023

elasticmachine commented Dec 8, 2023

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

mergify bot commented Jan 17, 2024

elasticmachine commented Jan 17, 2024

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh Jan 24, 2024

Choose a reason for hiding this comment

efd6 Jan 24, 2024

Choose a reason for hiding this comment

elasticmachine commented Jan 24, 2024

💔 Build Failed

Build stats

Pipeline error

❕ Flaky test report

🤖 GitHub comments

elasticmachine commented Jan 25, 2024

💚 Build Succeeded

Build stats

❕ Flaky test report

efd6 commented Dec 5, 2023 •

edited

Loading

`Error signal`