migrate x-pack/libbeat to static #38964

sharbuz · 2024-04-16T09:24:51Z

What is the problem this PR solves?

Jenkins->Buildkite pipelines migration

Migrate x-pack/libbeat from dynamic to static pipeline

Examples of the pipelines:
https://buildkite.com/elastic/beats/builds/5828
https://buildkite.com/elastic/beats-xpack-libbeat/builds/2673

Related issues

https://github.com/elastic/ingest-dev/issues/1693

botelastic · 2024-04-16T09:25:00Z

This pull request doesn't have a Team:<team> label.

mergify · 2024-04-16T09:25:32Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b migrate-xpack-libbeat-to-static upstream/migrate-xpack-libbeat-to-static
git merge upstream/main
git push upstream migrate-xpack-libbeat-to-static

elasticmachine · 2024-04-16T09:37:56Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 13 min 11 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2024-04-16T11:26:36Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b migrate-xpack-libbeat-to-static upstream/migrate-xpack-libbeat-to-static
git merge upstream/main
git push upstream migrate-xpack-libbeat-to-static

dliappis

Mostly good, but there are a few things missing for the migration.
For testing, can you enable the :Windows label to also see the windows extended tests results?

.buildkite/scripts/common.sh

.buildkite/x-pack/pipeline.xpack.libbeat.yml

.buildkite/pipeline.yml

.buildkite/x-pack/pipeline.xpack.libbeat.yml

mergify · 2024-04-17T15:20:40Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b migrate-xpack-libbeat-to-static upstream/migrate-xpack-libbeat-to-static
git merge upstream/main
git push upstream migrate-xpack-libbeat-to-static

…beats into migrate-xpack-libbeat-to-static

.buildkite/scripts/common.sh

dliappis

We are getting there. I think there were some accidental additions for x-pack/heartbeat and x-pack/filebeat

.buildkite/scripts/common.sh

.buildkite/x-pack/pipeline.xpack.libbeat.yml

.buildkite/scripts/common.sh

dliappis · 2024-04-19T10:01:25Z

.buildkite/scripts/common.sh

@@ -90,18 +89,12 @@ packaging_changeset=(
  )

 case "${BUILDKITE_PIPELINE_SLUG}" in
-  "beats-libbeat")
-    BEAT_CHANGESET_REFERENCE=${libbeat_changeset[@]}
-    ;;


are we intentionally removing beats-libbeat? the PR is about x-pack/libbeat

answered here

dliappis · 2024-04-19T10:02:06Z

.buildkite/scripts/common.sh

-libbeat_changeset=(
-  "^libbeat/.*"
-  )
-


is this on purpose? given that the PR is about x-pack/libbeat

libbeat pipeline has already merged.
Also, in this pipeline I've added one part from libbeat (we discussed that earlier):

beats/.buildkite/pipeline.yml

Line 268 in cd81faa

diff: "git diff --name-only origin/${GITHUB_PR_TARGET_BRANCH}...HEAD"

dliappis · 2024-04-19T10:06:18Z

.buildkite/pipeline.yml

+                - dev-tools/
+                - testing/
+                # x-pack
+                - libbeat/


My understanding from /Jenkinsfile.yml is that the common changesets for x-pack are:

# x-pack - go.mod - pytest.ini - dev-tools/ - libbeat/ - testing/ - x-pack/libbeat/

we've already used this pattern in x-pack/heartbeat so perhaps we can standardize i.e. replace both oss and x-pack with one "# x-pack" section for x-pack related projects (similar to Jenkinsfile)?

ok, let's compare what we have:

- x-pack/libbeat/ - .buildkite/x-pack/pipeline.xpack.libbeat.yml - .buildkite/scripts - .buildkite/hooks #OSS - go.mod - pytest.ini - dev-tools/ - testing/ # x-pack - libbeat/

or you mean we shouldn't merge them?

they are the same, I am just saying that for consistency's sake we should have a section

# x-pack - go.mod - pytest.ini - dev-tools/ - libbeat/ - testing/ - x-pack/libbeat/

for all x-pack pipelines. The pipeline specific bits will, of course, remain above, so the total thing with my proposal would be:

- x-pack/<whatever beat>/ # exceptionally here, since it's x-pack/libbeat which is part of the the x-pack common section below we should skip this line - .buildkite/x-pack/pipeline.xpack.libbeat.yml - .buildkite/scripts/ - .buildkite/hooks/ # x-pack - go.mod - pytest.ini - dev-tools/ - libbeat/ - testing/ - x-pack/libbeat/

Co-authored-by: Dimitrios Liappis <[email protected]>

…beats into migrate-xpack-libbeat-to-static

dliappis

LGTM

Left a minor ask in https://github.com/elastic/beats/pull/38964/files#r1572334321
which doesn't require another review cycle.

sharbuz · 2024-04-19T14:41:41Z

/test

* migrate x-pack/libbeat to static * Update pull-requests.json * change paths * Update pull-requests.json * change conditions for group-steps * change the pipeline.yml * work with remarks * work with remarks * remove libbeat and x-pack/libbeat changesets from the common.sh * remove libbeat and x-pack/libbeat changesets from the common.sh * fix merge issues * Update .buildkite/x-pack/pipeline.xpack.libbeat.yml Co-authored-by: Dimitrios Liappis <[email protected]> * apply common.sh from main and cleanup it * change paths --------- Co-authored-by: Dimitrios Liappis <[email protected]> (cherry picked from commit b0690b5) # Conflicts: # .buildkite/pipeline.yml # .buildkite/pull-requests.json # .buildkite/scripts/common.sh

* migrate x-pack/libbeat to static * Update pull-requests.json * change paths * Update pull-requests.json * change conditions for group-steps * change the pipeline.yml * work with remarks * work with remarks * remove libbeat and x-pack/libbeat changesets from the common.sh * remove libbeat and x-pack/libbeat changesets from the common.sh * fix merge issues * Update .buildkite/x-pack/pipeline.xpack.libbeat.yml Co-authored-by: Dimitrios Liappis <[email protected]> * apply common.sh from main and cleanup it * change paths --------- Co-authored-by: Dimitrios Liappis <[email protected]> (cherry picked from commit b0690b5) # Conflicts: # .buildkite/pipeline.yml # .buildkite/pull-requests.json # .buildkite/scripts/common.sh # .buildkite/x-pack/pipeline.xpack.libbeat.yml

* migrate x-pack/libbeat to static * Update pull-requests.json * change paths * Update pull-requests.json * change conditions for group-steps * change the pipeline.yml * work with remarks * work with remarks * remove libbeat and x-pack/libbeat changesets from the common.sh * remove libbeat and x-pack/libbeat changesets from the common.sh * fix merge issues * Update .buildkite/x-pack/pipeline.xpack.libbeat.yml Co-authored-by: Dimitrios Liappis <[email protected]> * apply common.sh from main and cleanup it * change paths --------- Co-authored-by: Dimitrios Liappis <[email protected]> (cherry picked from commit b0690b5) # Conflicts: # .buildkite/pipeline.yml # .buildkite/scripts/common.sh

* migrate x-pack/libbeat to static (#38964) * migrate x-pack/libbeat to static * Update pull-requests.json * change paths * Update pull-requests.json * change conditions for group-steps * change the pipeline.yml * work with remarks * work with remarks * remove libbeat and x-pack/libbeat changesets from the common.sh * remove libbeat and x-pack/libbeat changesets from the common.sh * fix merge issues * Update .buildkite/x-pack/pipeline.xpack.libbeat.yml Co-authored-by: Dimitrios Liappis <[email protected]> * apply common.sh from main and cleanup it * change paths --------- Co-authored-by: Dimitrios Liappis <[email protected]> (cherry picked from commit b0690b5) # Conflicts: # .buildkite/pipeline.yml # .buildkite/pull-requests.json # .buildkite/scripts/common.sh * Update pipeline.yml * Update pull-requests.json * Update pull-requests.json * Update common.sh * Update common.sh --------- Co-authored-by: Siarhei Harbuz <[email protected]>

* migrate x-pack/libbeat to static (#38964) * migrate x-pack/libbeat to static * Update pull-requests.json * change paths * Update pull-requests.json * change conditions for group-steps * change the pipeline.yml * work with remarks * work with remarks * remove libbeat and x-pack/libbeat changesets from the common.sh * remove libbeat and x-pack/libbeat changesets from the common.sh * fix merge issues * Update .buildkite/x-pack/pipeline.xpack.libbeat.yml Co-authored-by: Dimitrios Liappis <[email protected]> * apply common.sh from main and cleanup it * change paths --------- Co-authored-by: Dimitrios Liappis <[email protected]> (cherry picked from commit b0690b5) # Conflicts: # .buildkite/pipeline.yml # .buildkite/scripts/common.sh * Update pipeline.yml * Update common.sh * Update common.sh --------- Co-authored-by: Siarhei Harbuz <[email protected]>

* migrate x-pack/libbeat to static (#38964) * migrate x-pack/libbeat to static * Update pull-requests.json * change paths * Update pull-requests.json * change conditions for group-steps * change the pipeline.yml * work with remarks * work with remarks * remove libbeat and x-pack/libbeat changesets from the common.sh * remove libbeat and x-pack/libbeat changesets from the common.sh * fix merge issues * Update .buildkite/x-pack/pipeline.xpack.libbeat.yml Co-authored-by: Dimitrios Liappis <[email protected]> * apply common.sh from main and cleanup it * change paths --------- Co-authored-by: Dimitrios Liappis <[email protected]> (cherry picked from commit b0690b5) # Conflicts: # .buildkite/pipeline.yml # .buildkite/pull-requests.json # .buildkite/scripts/common.sh # .buildkite/x-pack/pipeline.xpack.libbeat.yml * Update pipeline.yml * Update pull-requests.json * Update pull-requests.json * Update common.sh * Update pipeline.xpack.libbeat.yml * Update pipeline.xpack.libbeat.yml * Update pipeline.xpack.libbeat.yml --------- Co-authored-by: Siarhei Harbuz <[email protected]>

migrate x-pack/libbeat to static

1ecc481

sharbuz added libbeat arm Enable builds in the CI for ARM testing backport-7.17 Automated backport to the 7.17 branch with mergify backport-v8.13.0 Automated backport with mergify labels Apr 16, 2024

sharbuz self-assigned this Apr 16, 2024

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 16, 2024

sharbuz and others added 2 commits April 16, 2024 12:25

Update pull-requests.json

1c2d56a

change paths

0a2a955

Merge branch 'main' into migrate-xpack-libbeat-to-static

0196f76

sharbuz added 2 commits April 16, 2024 16:46

Merge branch 'main' into migrate-xpack-libbeat-to-static

e9494c5

Update pull-requests.json

98879df

sharbuz marked this pull request as ready for review April 16, 2024 14:02

sharbuz requested a review from a team as a code owner April 16, 2024 14:02

Siarhei Harbuz added 2 commits April 16, 2024 17:23

change conditions for group-steps

b964c7d

change the pipeline.yml

6a5b507

sharbuz requested review from dliappis and alexsapran April 17, 2024 08:07

dliappis requested changes Apr 17, 2024

View reviewed changes

work with remarks

976c177

dliappis self-requested a review April 19, 2024 08:38

Siarhei Harbuz and others added 4 commits April 19, 2024 11:47

work with remarks

ad23254

Merge branch 'main' into migrate-xpack-libbeat-to-static

64a37ca

remove libbeat and x-pack/libbeat changesets from the common.sh

90e465f

Merge branch 'migrate-xpack-libbeat-to-static' of github.com:sharbuz/…

93c4c4e

…beats into migrate-xpack-libbeat-to-static

remove libbeat and x-pack/libbeat changesets from the common.sh

cd81faa

alexsapran reviewed Apr 19, 2024

View reviewed changes

.buildkite/scripts/common.sh Outdated Show resolved Hide resolved

alexsapran reviewed Apr 19, 2024

View reviewed changes

.buildkite/scripts/common.sh Outdated Show resolved Hide resolved

dliappis requested changes Apr 19, 2024

View reviewed changes

Siarhei Harbuz and others added 4 commits April 19, 2024 13:42

fix merge issues

08a8969

Update .buildkite/x-pack/pipeline.xpack.libbeat.yml

ae6fb75

Co-authored-by: Dimitrios Liappis <[email protected]>

apply common.sh from main and cleanup it

ca82305

Merge branch 'migrate-xpack-libbeat-to-static' of github.com:sharbuz/…

6782b90

…beats into migrate-xpack-libbeat-to-static

dliappis approved these changes Apr 19, 2024

View reviewed changes

Siarhei Harbuz and others added 2 commits April 19, 2024 16:04

change paths

ff1e84e

Merge branch 'main' into migrate-xpack-libbeat-to-static

732aca1

sharbuz removed libbeat arm Enable builds in the CI for ARM testing labels Apr 19, 2024

sharbuz merged commit b0690b5 into elastic:main Apr 19, 2024
77 of 110 checks passed

This was referenced Apr 19, 2024

[7.17](backport #38964) migrate x-pack/libbeat to static #39082

Merged

[8.13](backport #38964) migrate x-pack/libbeat to static #39083

Merged

sharbuz added the backport-v8.14.0 Automated backport with mergify label Apr 19, 2024

mergify bot mentioned this pull request Apr 19, 2024

[8.14](backport #38964) migrate x-pack/libbeat to static #39086

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

migrate x-pack/libbeat to static #38964

migrate x-pack/libbeat to static #38964

sharbuz commented Apr 16, 2024 •

edited

Loading

botelastic bot commented Apr 16, 2024

mergify bot commented Apr 16, 2024

elasticmachine commented Apr 16, 2024 •

edited

Loading

Build stats

mergify bot commented Apr 16, 2024

dliappis left a comment

mergify bot commented Apr 17, 2024

dliappis left a comment

dliappis Apr 19, 2024

sharbuz Apr 19, 2024

dliappis Apr 19, 2024

sharbuz Apr 19, 2024

dliappis Apr 19, 2024

sharbuz Apr 19, 2024 •

edited

Loading

sharbuz Apr 19, 2024 •

edited

Loading

dliappis Apr 19, 2024 •

edited

Loading

dliappis left a comment

sharbuz commented Apr 19, 2024

migrate x-pack/libbeat to static #38964

migrate x-pack/libbeat to static #38964

Conversation

sharbuz commented Apr 16, 2024 • edited Loading

What is the problem this PR solves?

Related issues

botelastic bot commented Apr 16, 2024

mergify bot commented Apr 16, 2024

elasticmachine commented Apr 16, 2024 • edited Loading

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

mergify bot commented Apr 16, 2024

dliappis left a comment

Choose a reason for hiding this comment

mergify bot commented Apr 17, 2024

dliappis left a comment

Choose a reason for hiding this comment

dliappis Apr 19, 2024

Choose a reason for hiding this comment

sharbuz Apr 19, 2024

Choose a reason for hiding this comment

dliappis Apr 19, 2024

Choose a reason for hiding this comment

sharbuz Apr 19, 2024

Choose a reason for hiding this comment

dliappis Apr 19, 2024

Choose a reason for hiding this comment

sharbuz Apr 19, 2024 • edited Loading

Choose a reason for hiding this comment

sharbuz Apr 19, 2024 • edited Loading

Choose a reason for hiding this comment

dliappis Apr 19, 2024 • edited Loading

Choose a reason for hiding this comment

dliappis left a comment

Choose a reason for hiding this comment

sharbuz commented Apr 19, 2024

sharbuz commented Apr 16, 2024 •

edited

Loading

elasticmachine commented Apr 16, 2024 •

edited

Loading

sharbuz Apr 19, 2024 •

edited

Loading

sharbuz Apr 19, 2024 •

edited

Loading

dliappis Apr 19, 2024 •

edited

Loading