Add Integration tests to Auditbeat CI pipeline #37892

mjwolf · 2024-02-06T19:02:51Z

Proposed commit message

Refactor the CI integration test steps for the Auditbeat pipeline to use build matrices for Linux tests. By doing this, it will be easier to add additional VM types to the integration tests in the future.

With this, the integration tests will run on Ubuntu 20.04, 22.04, 24.04 for both x86_64 and arm64, and RHEL 9 x86_64.

Checklist

My code follows the style guidelines of this project
~~I have commented my code, particularly in hard-to-understand areas~~
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
~~I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

mergify · 2024-02-06T19:04:38Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @mjwolf? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2024-02-06T19:09:59Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2024-02-07T22:21:13.971+0000
Duration: 12 min 27 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3
Skipped	0
Total	3

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-02-06T22:25:57Z

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

.buildkite/auditbeat/auditbeat-pipeline.yml

.buildkite/auditbeat/scripts/integ-tests.sh

mmat11 · 2024-02-06T22:41:37Z

I guess these can be removed then https://github.com/elastic/beats/blob/main/auditbeat/Jenkinsfile.yml#L33-L50

.buildkite/auditbeat/auditbeat-pipeline.yml

dliappis

Looks good, thank you!

Left a comment to raise awareness on GH status notifications and that by using a matrix, failures will be collapsed under one entry. If you are ok with this I won't object, just wanted to make sure the behavior is clearly understood.

dliappis · 2024-06-04T06:38:47Z

.buildkite/auditbeat/auditbeat-pipeline.yml

          machineType: "${GCP_DEFAULT_MACHINE_TYPE}"
        artifact_paths:
          - "auditbeat/build/*.xml"
          - "auditbeat/build/*.json"
        notify:
          - github_commit_status:
-              context: "auditbeat: Linux Integration Tests"
+              context: "auditbeat: Linux x86_64 Integration Tests"


By using a matrix and having a static context we'll be facing the issue that all failures will be collapsed under the same GH status notification.

For example, right now, with passing tests, the notification looks like:

If we click the link it takes us to:

which represents both Auditbeat Linux x86_64 Integration Tests -- family/platform-ingest-beats-ubuntu-2004 and Auditbeat Linux x86_64 Integration Tests -- family/platform-ingest-beats-ubuntu-2204

So if there is a failure, clicking on the link will take the user to the top of the matrix section, not the specific failed job. Looking at the official BK docs it might be impossible to add {{matrix.image}} -- or another additional property like {{matrix.name}} -- under notify: ["github_commit_stats": {"context": "the message ... {{matrix.property}}"}] (maybe worth testing to be 100% sure).

In any case, my main comment here is to raise awareness about the behavior and that with failures it won't be able to quickly see in the status section which specific image failed.

dliappis · 2024-06-04T06:41:59Z

.buildkite/auditbeat/auditbeat-pipeline.yml

+  AWS_IMAGE_UBUNTU_2004_ARM64: "platform-ingest-beats-ubuntu-2004-aarch64"
+  AWS_IMAGE_UBUNTU_2204_ARM64: "platform-ingest-beats-ubuntu-2204-aarch64"
+
+  IMAGE_MACOS_ARM64: "generic-13-ventura-arm"


nit: re: renaming IMAGE_MACOS_ARM to IMAGE_MACOS_ARM64, ideally we do this in a future PR where we change this as a convention for all the pipelines. For now, it might be ok to leave the name as is, given that there is no 32bit ARM architecture for macOS?

mergify · 2024-09-11T13:10:21Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b auditbeat-integTest-ci upstream/auditbeat-integTest-ci
git merge upstream/main
git push upstream auditbeat-integTest-ci

elasticmachine · 2024-09-16T13:10:57Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

With auditbeat, there can be differences in Linux kernels and system architectures that can affect behaviour. This adds additional system types to the integration tests that will test more of these configurations.

mjwolf · 2024-10-18T19:35:56Z

I'd like to revive this PR. I've added all the Linux 'platform-ingest-beats' VM images that currently exist and removed the variable name changes that weren't necessary.

@leehinman, @dliappis, @pkoutsovasilis could you review this again, since it's changed since you last looked at it?

rowlandgeoff · 2024-10-24T21:17:58Z

LGTM. @oakrizan can you please review? Any overlap with your current changes?

oakrizan · 2024-10-25T07:13:40Z

We deliberately were avoiding matrix usage. In addition to what Dimitrios said about matrix - if one of the matrix-based steps fails, then particular GH check will hang in "Pending" state.

oakrizan

lgtm

Refactor the CI integration test steps for the Auditbeat pipeline to use build matrices for Linux tests. By doing this, it will be easier to add additional VM types to the integration tests in the future. With this, the integration tests will run on Ubuntu 20.04, 22.04, 24.04 for both x86_64 and arm64, and RHEL 9 x86_64. (cherry picked from commit efbc4ff)

Refactor the CI integration test steps for the Auditbeat pipeline to use build matrices for Linux tests. By doing this, it will be easier to add additional VM types to the integration tests in the future. With this, the integration tests will run on Ubuntu 20.04, 22.04, 24.04 for both x86_64 and arm64, and RHEL 9 x86_64. (cherry picked from commit efbc4ff) Co-authored-by: Michael Wolf <[email protected]> Co-authored-by: Andrew Kroh <[email protected]>

Refactor the CI integration test steps for the Auditbeat pipeline to use build matrices for Linux tests. By doing this, it will be easier to add additional VM types to the integration tests in the future. With this, the integration tests will run on Ubuntu 20.04, 22.04, 24.04 for both x86_64 and arm64, and RHEL 9 x86_64. (cherry picked from commit efbc4ff) Co-authored-by: Michael Wolf <[email protected]>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 6, 2024

mergify bot assigned mjwolf Feb 6, 2024

mjwolf added the Team:Security-Deployment and Devices Deployment and Devices Team in Security Solution label Feb 6, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 6, 2024

mjwolf added needs_team Indicates that the issue/PR needs a Team:* label backport-skip Skip notification from the automated backport with mergify labels Feb 6, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 6, 2024

mjwolf added the enhancement label Feb 6, 2024

mjwolf force-pushed the auditbeat-integTest-ci branch from db2bb07 to 622fdfe Compare February 6, 2024 19:07

mjwolf force-pushed the auditbeat-integTest-ci branch 3 times, most recently from d98e7d0 to b42246a Compare February 6, 2024 21:26

mjwolf added backport-v8.12.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Feb 6, 2024

mjwolf force-pushed the auditbeat-integTest-ci branch 2 times, most recently from e1f1adc to 5d64714 Compare February 6, 2024 22:23

mjwolf marked this pull request as ready for review February 6, 2024 22:25

mjwolf requested a review from a team as a code owner February 6, 2024 22:25

mjwolf requested review from ycombinator and leehinman February 6, 2024 22:25

mjwolf requested review from oakrizan and mmat11 February 6, 2024 22:27

mmat11 reviewed Feb 6, 2024

View reviewed changes

.buildkite/auditbeat/auditbeat-pipeline.yml Outdated Show resolved Hide resolved

mmat11 reviewed Feb 6, 2024

View reviewed changes

.buildkite/auditbeat/scripts/integ-tests.sh Outdated Show resolved Hide resolved

mmat11 reviewed Feb 6, 2024

View reviewed changes

.buildkite/auditbeat/auditbeat-pipeline.yml Outdated Show resolved Hide resolved

mmat11 reviewed Feb 6, 2024

View reviewed changes

.buildkite/auditbeat/auditbeat-pipeline.yml Outdated Show resolved Hide resolved

dliappis reviewed Jun 4, 2024

View reviewed changes

v1v added the backport-8.x Automated backport to the 8.x branch with mergify label Sep 11, 2024

pierrehilbert added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Sep 16, 2024

leehinman approved these changes Oct 17, 2024

View reviewed changes

Add additional integration test systems to Auditbeat CI

15d5be4

With auditbeat, there can be differences in Linux kernels and system architectures that can affect behaviour. This adds additional system types to the integration tests that will test more of these configurations.

mjwolf force-pushed the auditbeat-integTest-ci branch from 023021a to 15d5be4 Compare October 18, 2024 18:28

mjwolf added backport-8.16 Automated backport with mergify and removed macOS Enable builds in the CI for darwin testing backport-v8.12.0 Automated backport with mergify labels Oct 18, 2024

fix typo

09b06b8

mjwolf requested review from dliappis, pkoutsovasilis and leehinman October 18, 2024 19:36

leehinman approved these changes Oct 18, 2024

View reviewed changes

Merge branch 'main' into auditbeat-integTest-ci

370d738

mjwolf enabled auto-merge (squash) October 21, 2024 20:48

mjwolf requested a review from a team October 21, 2024 21:04

oakrizan approved these changes Oct 25, 2024

View reviewed changes

mjwolf merged commit efbc4ff into elastic:main Oct 25, 2024
24 checks passed

This was referenced Oct 25, 2024

[8.16](backport #37892) Add Integration tests to Auditbeat CI pipeline #41450

Merged

[8.x](backport #37892) Add Integration tests to Auditbeat CI pipeline #41451

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Integration tests to Auditbeat CI pipeline #37892

Add Integration tests to Auditbeat CI pipeline #37892

mjwolf commented Feb 6, 2024 •

edited

Loading

mergify bot commented Feb 6, 2024

elasticmachine commented Feb 6, 2024 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Feb 6, 2024

mmat11 commented Feb 6, 2024

dliappis left a comment

dliappis Jun 4, 2024 •

edited

Loading

dliappis Jun 4, 2024

mergify bot commented Sep 11, 2024

elasticmachine commented Sep 16, 2024

mjwolf commented Oct 18, 2024

rowlandgeoff commented Oct 24, 2024

oakrizan commented Oct 25, 2024

oakrizan left a comment

Add Integration tests to Auditbeat CI pipeline #37892

Add Integration tests to Auditbeat CI pipeline #37892

Conversation

mjwolf commented Feb 6, 2024 • edited Loading

Proposed commit message

Checklist

mergify bot commented Feb 6, 2024

elasticmachine commented Feb 6, 2024 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

elasticmachine commented Feb 6, 2024

mmat11 commented Feb 6, 2024

dliappis left a comment

Choose a reason for hiding this comment

dliappis Jun 4, 2024 • edited Loading

Choose a reason for hiding this comment

dliappis Jun 4, 2024

Choose a reason for hiding this comment

mergify bot commented Sep 11, 2024

elasticmachine commented Sep 16, 2024

mjwolf commented Oct 18, 2024

rowlandgeoff commented Oct 24, 2024

oakrizan commented Oct 25, 2024

oakrizan left a comment

Choose a reason for hiding this comment

mjwolf commented Feb 6, 2024 •

edited

Loading

elasticmachine commented Feb 6, 2024 •

edited

Loading

dliappis Jun 4, 2024 •

edited

Loading