Migrate xpack packetbeat #38135

sharbuz · 2024-02-26T13:37:59Z

What is the problem this PR solves?

Jenkins->Buildkite pipelines migration

Migrate x-pack packetbeat pipeline

Example of the pipelines:
beats-xpack-packetbeat
Examples of the affected pipelines:
beats-libbeat
beats-metricbeat
beats-packetbeat
beats-winlogbeat
beats-xpack-libbeat
beats-xpack-metricbeat

Related issues

https://github.com/elastic/ingest-dev/issues/1693
#38142 - related to commented-out Windows system tests

v1v · 2024-03-13T10:08:59Z

.buildkite/scripts/win_unit_tests.ps1

+        mage build unitTest
+    }
+}
+elseif ($testType -eq "systemtest") {


The name of the file .buildkite/scripts/win_unit_tests.ps1 is specific about unit_tests while it also runs mage systemTest.

What do you think about this proposal that can avoid maintaining the .buildkite/scripts/win_unit_tests.ps1 but the pipeline itself?

rename the file for buildkite/scripts/win_mage.ps1, it's more generic and could accept different mage goals.

testType to be renamed for mageGoals and then simplify the script with the below snippet:

if ($mageGoals -eq "systemTest") { google_cloud_auth } mage $mageGoals

Change the pipeline stages with:

# .buildkite/scripts/generate_xpack_packetbeat_pipeline.sh - label: ":windows: Windows Unit Tests - {{matrix.image}}" command: ".buildkite/scripts/win_mage.ps1 build unitTest" ... ## TODO: uncomment when the issue https://github.com/elastic/beats/issues/38142 is solved # - label: ":windows: Windows 10 System Tests" # key: "extended-win-10-system-tests" #. command: ".buildkite/scripts/win_mage.ps1 systemTest"

# .buildkite/scripts/generate_xpack_ libbeat_pipeline.sh - label: ":windows: Windows 11 Unit Tests" command: ".buildkite/scripts/win_mage.ps1 -w reader/etw build goUnitTest"

I agree with you that it could be done as you suggest. There is only one remark. We gonna decline to use Windows scripts soon (I hope in 1 week). If it's possible to avoid additional changes right now, let's do that. @v1v WDYT?

Do you mean the file .buildkite/scripts/win_unit_tests.ps1 will not be used anymore soon? I still think the name of the file is not correct for the nature of what it does, so .buildkite/scripts/win_tests.ps1 could be more meaningful.

Correct.
We are going to use our own BK Windows images and @dliappis is working on it.
When it's ready we are going to use command steps with 1-3 commands without using PowerShell scripts.

Renaming this script requires a lot of changes in the already prepared dynamic pipelines.

I'll then wait for @dliappis 's feedback; if he has no concerns, I'm happy with this.

Thanks @v1v for the great feedback (as usual!) The custom Windows images are now ready, so I am good with letting this as is and get rid of many custom scripts and snippets, including the one discussed here, ASAP.

dliappis

LGTM thanks all for the great work and review discussions!

dliappis · 2024-03-13T12:54:42Z

.buildkite/scripts/win_unit_tests.ps1

+        mage build unitTest
+    }
+}
+elseif ($testType -eq "systemtest") {


Thanks @v1v for the great feedback (as usual!) The custom Windows images are now ready, so I am good with letting this as is and get rid of many custom scripts and snippets, including the one discussed here, ASAP.

dliappis · 2024-03-13T12:58:16Z

x-pack/packetbeat/magefile.go

@@ -172,6 +172,13 @@ func SystemTest(ctx context.Context) error {
 	return devtools.GoTest(ctx, args)
 }

+func getBucketName() string {


@sharbuz if we haven't it yet, let's create a post migration ticket in the usual repo and link it to https://github.com/elastic/ingest-dev/issues/2794 to remind us that the conditional behavior here should be removed once Jenkins has been sunset (I realize that it should be possible to run the tests locally by devs too but I believe they have access to it already).

done.
https://github.com/elastic/ingest-dev/issues/3006

jlind23 · 2024-03-13T15:52:32Z

Hey @nfritts @norrietaylor, your team approval is required here even though this is purely a CI change that we own.
Can we please get your thumbs up?

elasticmachine · 2024-03-14T05:05:35Z

💔 Build Failed

Buildkite Build
Commit: 5aa00de

Failed CI Steps

History

💔 Build #2394 failed 9585c26
💔 Build #2370 failed 37b956a
💔 Build #2365 failed e2e12d1
💔 Build #2363 failed 2954618

cc @sharbuz

elasticmachine · 2024-03-14T05:08:09Z

💔 Build Failed

Buildkite Build
Commit: 5aa00de

Failed CI Steps

History

💔 Build #3735 failed 9585c26
💔 Build #3711 failed 37b956a
💔 Build #3706 failed e2e12d1
💔 Build #3701 failed b23146a

cc @sharbuz

elasticmachine · 2024-03-14T05:20:22Z

💚 Build Succeeded

Buildkite Build
Commit: 5aa00de

History

💚 Build #2097 succeeded 9585c26
💚 Build #2073 succeeded 37b956a
💚 Build #2068 succeeded e2e12d1
💚 Build #2063 succeeded b23146a
💚 Build #2058 succeeded 31358c9
💚 Build #2044 succeeded 554c3a5

cc @sharbuz

elasticmachine · 2024-03-14T05:36:02Z

💚 Build Succeeded

Buildkite Build
Commit: 5aa00de

History

💚 Build #2104 succeeded 9585c26
💔 Build #2080 failed 37b956a
💚 Build #2075 succeeded e2e12d1
💔 Build #2073 failed 2954618
💛 Build #2051 was flaky 554c3a5
💚 Build #2022 succeeded 1a839aa

cc @sharbuz

elasticmachine · 2024-03-14T05:39:19Z

💚 Build Succeeded

Buildkite Build
Commit: 5aa00de

History

💚 Build #1260 succeeded 9585c26
💚 Build #1236 succeeded 37b956a
💚 Build #1231 succeeded e2e12d1
💚 Build #1207 succeeded 554c3a5
💚 Build #1178 succeeded 1a839aa
💚 Build #867 succeeded 2a1eb07

cc @sharbuz

elasticmachine · 2024-03-14T05:47:14Z

💚 Build Succeeded

Buildkite Build
Commit: 5aa00de

History

💚 Build #3311 succeeded 9585c26
💚 Build #3287 succeeded 37b956a
💔 Build #3282 failed e2e12d1
💚 Build #3258 succeeded 554c3a5
💔 Build #3229 failed 1a839aa
💚 Build #2918 succeeded 2a1eb07

cc @sharbuz

jlind23 · 2024-03-14T08:05:49Z

Force merging this as it is blocked due to unrelated problems, no MacOS workers are available to execute the job

* migrate xpack-metricbeat --------- Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Julien Lind <[email protected]> (cherry picked from commit e0d5fe5) # Conflicts: # .buildkite/scripts/setenv.sh # x-pack/packetbeat/magefile.go

* migrate xpack-metricbeat --------- Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Julien Lind <[email protected]> (cherry picked from commit e0d5fe5)

* migrate xpack-metricbeat --------- Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Julien Lind <[email protected]> (cherry picked from commit e0d5fe5) Co-authored-by: sharbuz <[email protected]>

Siarhei Harbuz and others added 30 commits February 21, 2024 12:55

migrate xpack-metricbeat

2b5dd7b

change pre-command

cf44101

add mage_version

e432525

add mage_version

0783eee

disable some trigger steps

2a1de26

change command with mage

7e55863

add ASDF_MAGE_VERSION

cf55e5b

fix arm image name

def5673

fix macos

85bfec8

test arm packaging

5af9510

test aws/macos

90bec63

test macos

bd038c2

test macos

77b29b2

test macos

5b41d3b

test macos

2751e38

test macos

88800c8

fix macos declare

4805ccb

add export MODULE

e242568

test aws

d41b68d

test aws

5319753

add withModule

32229ce

move withModule to pre-command

a09803e

move withModule to pre-command

271c788

move withModule to command

cd2282d

fix withModule

8eba9dc

test aws changes

1781fc0

test aws changes

c12fe5a

change the are_changed_only_paths

dd1e93e

revert aws changes and test

d0f433b

Merge branch 'main' into migrate-xpack-metricbeat

751403b

v1v reviewed Mar 13, 2024

View reviewed changes

Siarhei Harbuz added 4 commits March 13, 2024 12:42

test removing gcp credentials

b23146a

add gcp credentials

2954618

add gcp credentials

e2e12d1

remove debug and cleanup

37b956a

dliappis approved these changes Mar 13, 2024

View reviewed changes

v1v approved these changes Mar 13, 2024

View reviewed changes

sharbuz enabled auto-merge (squash) March 13, 2024 14:13

Merge branch 'main' into migrate-xpack-packetbeat

9585c26

mjwolf approved these changes Mar 13, 2024

View reviewed changes

Merge branch 'main' into migrate-xpack-packetbeat

5aa00de

jlind23 disabled auto-merge March 14, 2024 08:04

jlind23 merged commit e0d5fe5 into elastic:main Mar 14, 2024
38 of 46 checks passed

mergify bot mentioned this pull request Mar 14, 2024

[7.17](backport #38135) Migrate xpack packetbeat #38313

Merged

mergify bot pushed a commit that referenced this pull request Mar 14, 2024

Migrate xpack packetbeat (#38135)

63b8aaa

* migrate xpack-metricbeat --------- Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Julien Lind <[email protected]> (cherry picked from commit e0d5fe5)

mergify bot pushed a commit that referenced this pull request Mar 14, 2024

Migrate xpack packetbeat (#38135)

f4808a9

* migrate xpack-metricbeat --------- Co-authored-by: Victor Martinez <[email protected]> Co-authored-by: Julien Lind <[email protected]> (cherry picked from commit e0d5fe5)

This was referenced Mar 14, 2024

[8.12](backport #38135) Migrate xpack packetbeat #38314

Merged

[8.13](backport #38135) Migrate xpack packetbeat #38315

Merged

jlind23 pushed a commit that referenced this pull request Mar 18, 2024

[7.17](backport #38135) Migrate xpack packetbeat (#38313)

2352a39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate xpack packetbeat #38135

Migrate xpack packetbeat #38135

sharbuz commented Feb 26, 2024 •

edited

Loading

v1v Mar 13, 2024

sharbuz Mar 13, 2024

v1v Mar 13, 2024

sharbuz Mar 13, 2024

sharbuz Mar 13, 2024

v1v Mar 13, 2024

dliappis Mar 13, 2024

dliappis left a comment

dliappis Mar 13, 2024

dliappis Mar 13, 2024

sharbuz Mar 13, 2024

jlind23 commented Mar 13, 2024

elasticmachine commented Mar 14, 2024 •

edited

Loading

elasticmachine commented Mar 14, 2024 •

edited

Loading

elasticmachine commented Mar 14, 2024

elasticmachine commented Mar 14, 2024

elasticmachine commented Mar 14, 2024

elasticmachine commented Mar 14, 2024

jlind23 commented Mar 14, 2024

Migrate xpack packetbeat #38135

Migrate xpack packetbeat #38135

Conversation

sharbuz commented Feb 26, 2024 • edited Loading

What is the problem this PR solves?

Related issues

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dliappis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlind23 commented Mar 13, 2024

elasticmachine commented Mar 14, 2024 • edited Loading

💔 Build Failed

Failed CI Steps

History

elasticmachine commented Mar 14, 2024 • edited Loading

💔 Build Failed

Failed CI Steps

History

elasticmachine commented Mar 14, 2024

💚 Build Succeeded

History

elasticmachine commented Mar 14, 2024

💚 Build Succeeded

History

elasticmachine commented Mar 14, 2024

💚 Build Succeeded

History

elasticmachine commented Mar 14, 2024

💚 Build Succeeded

History

jlind23 commented Mar 14, 2024

sharbuz commented Feb 26, 2024 •

edited

Loading

elasticmachine commented Mar 14, 2024 •

edited

Loading

elasticmachine commented Mar 14, 2024 •

edited

Loading