migrate packetbeat pipeline #37825

sharbuz · 2024-02-01T16:14:05Z

What is the problem this PR solves?

Jenkins->Buildkite pipelines migration

How does this PR solve the problem?

Migrates the packetbeat pipeline
https://buildkite.com/elastic/beats-packetbeat/builds?branch=sharbuz%3Amigrate-packetbeat-pipeline

Example of the pipeline:
https://buildkite.com/elastic/beats-packetbeat/builds/1081
Examples of dependencies:
https://buildkite.com/elastic/beats-metricbeat/builds/2310
https://buildkite.com/elastic/beats-libbeat/builds/1083

Related issues

https://github.com/elastic/ingest-dev/issues/1693

botelastic · 2024-02-01T16:14:12Z

This pull request doesn't have a Team:<team> label.

elasticmachine · 2024-02-01T16:29:07Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 15 min 32 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-02-01T17:06:22Z

❕ Build Aborted

Either there was a build timeout or someone aborted the build.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 13 min 10 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-02-01T17:13:31Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 73 min 42 sec

Pipeline error

This error is likely related to the pipeline itself. Click here
and then you will see the error (either incorrect syntax or an invalid configuration).

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2024-02-02T12:23:49Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b migrate-packetbeat-pipeline upstream/migrate-packetbeat-pipeline
git merge upstream/main
git push upstream migrate-packetbeat-pipeline

mergify · 2024-02-05T14:32:18Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b migrate-packetbeat-pipeline upstream/migrate-packetbeat-pipeline
git merge upstream/main
git push upstream migrate-packetbeat-pipeline

…at-pipeline

v1v · 2024-02-07T16:50:49Z

.buildkite/scripts/generate_packetbeat_pipeline.sh

+  echo "The conditions don't match to requirements for generating pipeline steps."
+  exit 1


what's the reason for exit 1? I'd expect it should do nothing

If the conditions don't meet the requirements the $pipelineName won't be created and the commands cat $pipelineName and buildkite-agent pipeline upload $pipelineName will return the error File not found. I used the exit 1:
to avoid additional conditions for the commands above
to avoid creating an empty file
to speed up the script if the changes don't meet the requirements
Probably, in this case, I could use exit 0, but I just wanted to highlight the empty pipeline, and maybe I just need to add buildkite-agent annotate option here. WDYT?

If the conditions don't meet the requirements the $pipelineName won't be created and the commands cat $pipelineName and buildkite-agent pipeline upload $pipelineName will return the error File not found.

I understand the rationale for reporting an error if cat or similar commands failed.

I understand the reason for filtering the filters defined in https://github.com/elastic/beats/blob/main/.buildkite/pull-requests.json.

However, I cannot understand the rationale for using a condition for the mandatory stages in the packetbeat pipeline for something other than doc changes.

Any PR touching the docs should not run the mandatory or any other stages unless a GitHub comment or similar UI interactions are used.

Is there any other condition for the mandatory stages that must be considered?

The condition related to the DOCS is contained in the main pipeline and it will be implemented a bit later. Right now we have to see all pipelines are GREEN whenever we change something to understand how current changes affect our previous job.

what's the reason for exit 1? I'd expect it should do nothing

BTW, I've changed to exit 0, in this case, if nothing changed - it's fine and should be GREEN, I think.

dliappis

Basically LGTM from me.
I left an ask for better handling of when we run buildkite-agent ... in common.sh to make manual reproductions easier.

dliappis · 2024-02-08T11:07:59Z

.buildkite/scripts/common.sh

@@ -10,34 +10,45 @@ GITHUB_PR_TRIGGER_COMMENT=${GITHUB_PR_TRIGGER_COMMENT:-""}
 ONLY_DOCS=${ONLY_DOCS:-"true"}
 runLibbeat="$(buildkite-agent meta-data get runLibbeat --default ${runLibbeat:-"false"})"
 runMetricbeat="$(buildkite-agent meta-data get runMetricbeat --default ${runMetricbeat:-"false"})"
+runPacketbeat="$(buildkite-agent meta-data get runPacketbeat --default ${runPacketbeat:-"false"})"


A little unrelated to this PR, but what we are doing here makes manual reproductions difficult (i.e. this command fails when run manually). Could we improve the conditional such that buildkite-agent meta-data get is run only if the env var runPackagebeat is undefined or not false? (applies to all declarations here).

For context: https://elastic.slack.com/archives/C0D8SUKB2/p1707380999484699?thread_ts=1707312641.347669&cid=C0D8SUKB2

This option is for the future main pipeline, where we will define the steps of the pipeline...
We have a similar variable in the main pipeline - runAllStages

.buildkite/scripts/common.sh

dliappis

LGTM

(cherry picked from commit e4756b2)

elasticmachine · 2024-02-08T17:56:41Z

💔 Build Failed

Buildkite Build
Commit: 4019ce6

Failed CI Steps

Package pipeline

History

💔 Build #2775 failed 9b3d894
💔 Build #2765 failed d6a798b
💔 Build #2753 failed c120fab
💔 Build #2736 failed 3ddb056
💔 Build #2677 failed fbc73d5

cc @sharbuz

(cherry picked from commit e4756b2) Co-authored-by: sharbuz <[email protected]>

migrate packetbeat pipeline

e54f671

sharbuz added Packetbeat macOS Enable builds in the CI for darwin testing labels Feb 1, 2024

sharbuz self-assigned this Feb 1, 2024

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 1, 2024

sharbuz added backport-7.17 Automated backport to the 7.17 branch with mergify backport-v8.12.0 Automated backport with mergify labels Feb 1, 2024

Siarhei Harbuz added 2 commits February 1, 2024 18:52

fix typo

d1ad060

add with_dependencies

0174a2a

sharbuz added the arm Enable builds in the CI for ARM testing label Feb 2, 2024

v1v mentioned this pull request Feb 5, 2024

fix: github API comment is different when modigying an existing one elastic/apm-pipeline-library#2520

Merged

Siarhei Harbuz and others added 13 commits February 5, 2024 17:29

Merge branch 'main' of github.com:sharbuz/beats into migrate-packetbe…

176e508

…at-pipeline

Merge branch 'main' into migrate-packetbeat-pipeline

22c5a83

run all pipeline's steps

9e88642

disable the failed Windows test and change pre-command

073b241

uncomment Windows 10 tests

3e25547

fir RHEL9 step

572dc15

fir RHEL9 step

64c2fe2

fix RHEL9

d35edb8

fix RHEL9

9ebe4c6

fix RHEL9

dda216e

fix RHEL9

8809bb7

fix RHEL9

386f384

Merge branch 'main' into migrate-packetbeat-pipeline

01b5bf1

sharbuz requested review from belimawr, rdner, dliappis, v1v, pazone and jlind23 February 7, 2024 16:38

v1v reviewed Feb 7, 2024

View reviewed changes

sharbuz requested a review from v1v February 8, 2024 06:59

Siarhei Harbuz added 3 commits February 8, 2024 11:25

change exit 1 to exit 0 for the dynamic pipeline

3ddb056

update dynamic pipeline

c120fab

fix VAR name

d6a798b

dliappis reviewed Feb 8, 2024

View reviewed changes

oakrizan mentioned this pull request Feb 8, 2024

Heartbeat pipeline migration #37911

Merged

6 tasks

v1v approved these changes Feb 8, 2024

View reviewed changes

Siarhei Harbuz added 3 commits February 8, 2024 15:13

change checkLinuxType

9b3d894

fix BK VARs

a6f7a12

fix BK VARs

4019ce6

dliappis self-requested a review February 8, 2024 17:18

dliappis approved these changes Feb 8, 2024

View reviewed changes

sharbuz enabled auto-merge (squash) February 8, 2024 17:19

jlind23 approved these changes Feb 8, 2024

View reviewed changes

jlind23 disabled auto-merge February 8, 2024 17:42

jlind23 merged commit e4756b2 into elastic:main Feb 8, 2024
27 of 34 checks passed

mergify bot pushed a commit that referenced this pull request Feb 8, 2024

migrate packetbeat pipeline (#37825)

8dce588

(cherry picked from commit e4756b2)

mergify bot pushed a commit that referenced this pull request Feb 8, 2024

migrate packetbeat pipeline (#37825)

30e1f5d

(cherry picked from commit e4756b2)

This was referenced Feb 8, 2024

[7.17](backport #37825) migrate packetbeat pipeline #37931

Merged

[8.12](backport #37825) migrate packetbeat pipeline #37932

Merged

sharbuz added a commit that referenced this pull request Feb 8, 2024

migrate packetbeat pipeline (#37825) (#37932)

b9fe81a

(cherry picked from commit e4756b2) Co-authored-by: sharbuz <[email protected]>

sharbuz added a commit that referenced this pull request Feb 8, 2024

migrate packetbeat pipeline (#37825) (#37931)

1f7fb12

(cherry picked from commit e4756b2) Co-authored-by: sharbuz <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

migrate packetbeat pipeline #37825

migrate packetbeat pipeline #37825

sharbuz commented Feb 1, 2024 •

edited

Loading

botelastic bot commented Feb 1, 2024

elasticmachine commented Feb 1, 2024

Build stats

elasticmachine commented Feb 1, 2024

Build stats

elasticmachine commented Feb 1, 2024 •

edited

Loading

Build stats

mergify bot commented Feb 2, 2024

mergify bot commented Feb 5, 2024

v1v Feb 7, 2024

sharbuz Feb 8, 2024 •

edited

Loading

v1v Feb 8, 2024

sharbuz Feb 8, 2024

sharbuz Feb 8, 2024

dliappis left a comment

dliappis Feb 8, 2024

sharbuz Feb 8, 2024

dliappis left a comment

elasticmachine commented Feb 8, 2024

		echo "The conditions don't match to requirements for generating pipeline steps."
		exit 1

migrate packetbeat pipeline #37825

migrate packetbeat pipeline #37825

Conversation

sharbuz commented Feb 1, 2024 • edited Loading

What is the problem this PR solves?

How does this PR solve the problem?

Related issues

botelastic bot commented Feb 1, 2024

elasticmachine commented Feb 1, 2024

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

elasticmachine commented Feb 1, 2024

❕ Build Aborted

Build stats

🤖 GitHub comments

elasticmachine commented Feb 1, 2024 • edited Loading

💔 Build Failed

Build stats

Pipeline error

❕ Flaky test report

🤖 GitHub comments

mergify bot commented Feb 2, 2024

mergify bot commented Feb 5, 2024

v1v Feb 7, 2024

Choose a reason for hiding this comment

sharbuz Feb 8, 2024 • edited Loading

Choose a reason for hiding this comment

v1v Feb 8, 2024

Choose a reason for hiding this comment

sharbuz Feb 8, 2024

Choose a reason for hiding this comment

sharbuz Feb 8, 2024

Choose a reason for hiding this comment

dliappis left a comment

Choose a reason for hiding this comment

dliappis Feb 8, 2024

Choose a reason for hiding this comment

sharbuz Feb 8, 2024

Choose a reason for hiding this comment

dliappis left a comment

Choose a reason for hiding this comment

elasticmachine commented Feb 8, 2024

💔 Build Failed

Failed CI Steps

History

sharbuz commented Feb 1, 2024 •

edited

Loading

elasticmachine commented Feb 1, 2024 •

edited

Loading

sharbuz Feb 8, 2024 •

edited

Loading