Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

migrate x-pack/libbeat to static #38964

Merged
merged 20 commits into from
Apr 19, 2024
Merged

migrate x-pack/libbeat to static #38964

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
1ecc481
migrate x-pack/libbeat to static
Apr 16, 2024
1c2d56a
Update pull-requests.json
sharbuz Apr 16, 2024
0a2a955
change paths
Apr 16, 2024
0196f76
Merge branch 'main' into migrate-xpack-libbeat-to-static
sharbuz Apr 16, 2024
e9494c5
Merge branch 'main' into migrate-xpack-libbeat-to-static
sharbuz Apr 16, 2024
98879df
Update pull-requests.json
sharbuz Apr 16, 2024
b964c7d
change conditions for group-steps
Apr 16, 2024
6a5b507
change the pipeline.yml
Apr 17, 2024
976c177
work with remarks
Apr 19, 2024
ad23254
work with remarks
Apr 19, 2024
64a37ca
Merge branch 'main' into migrate-xpack-libbeat-to-static
sharbuz Apr 19, 2024
90e465f
remove libbeat and x-pack/libbeat changesets from the common.sh
Apr 19, 2024
93c4c4e
Merge branch 'migrate-xpack-libbeat-to-static' of github.com:sharbuz/…
Apr 19, 2024
cd81faa
remove libbeat and x-pack/libbeat changesets from the common.sh
Apr 19, 2024
08a8969
fix merge issues
Apr 19, 2024
ae6fb75
Update .buildkite/x-pack/pipeline.xpack.libbeat.yml
sharbuz Apr 19, 2024
ca82305
apply common.sh from main and cleanup it
Apr 19, 2024
6782b90
Merge branch 'migrate-xpack-libbeat-to-static' of github.com:sharbuz/…
Apr 19, 2024
ff1e84e
change paths
Apr 19, 2024
732aca1
Merge branch 'main' into migrate-xpack-libbeat-to-static
sharbuz Apr 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .buildkite/hooks/pre-command
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,6 @@ ENABLED_BEATS_PIPELINES_SLUGS=(
"beats-packetbeat"
"beats-winlogbeat"
"beats-winlogbeat"
"beats-xpack-libbeat"
"beats-xpack-packetbeat"
"beats-xpack-winlogbeat"
"beats-xpack-dockerlogbeat"
Expand Down
30 changes: 29 additions & 1 deletion .buildkite/pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,7 @@ steps:
- label: "Trigger Libbeat"
plugins:
- monorepo-diff#v1.0.1:
diff: "git diff --name-only ${GITHUB_PR_TARGET_BRANCH}...HEAD"
diff: "git diff --name-only origin/${GITHUB_PR_TARGET_BRANCH}...HEAD"
watch:
- path:
- libbeat/
Expand All @@ -284,6 +284,34 @@ steps:
branch: "${BUILDKITE_BRANCH}"
env:
- BUILDKITE_PULL_REQUEST=${BUILDKITE_PULL_REQUEST}
- BUILDKITE_PULL_REQUEST_BASE_BRANCH=${BUILDKITE_PULL_REQUEST_BASE_BRANCH}
- GITHUB_PR_LABELS=${GITHUB_PR_LABELS}

- label: "Trigger x-pack/libbeat"
plugins:
- monorepo-diff#v1.0.1:
diff: "git diff --name-only origin/${GITHUB_PR_TARGET_BRANCH}...HEAD"
watch:
- path:
- x-pack/libbeat/
- .buildkite/x-pack/pipeline.xpack.libbeat.yml
- .buildkite/scripts
- .buildkite/hooks
#OSS
- go.mod
- pytest.ini
- dev-tools/
- testing/
# x-pack
- libbeat/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My understanding from /Jenkinsfile.yml is that the common changesets for x-pack are:

 # x-pack
                - go.mod
                - pytest.ini
                - dev-tools/
                - libbeat/
                - testing/
                - x-pack/libbeat/

we've already used this pattern in x-pack/heartbeat so perhaps we can standardize i.e. replace both oss and x-pack with one "# x-pack" section for x-pack related projects (similar to Jenkinsfile)?

Copy link
Contributor Author

@sharbuz sharbuz Apr 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, let's compare what we have:

                - x-pack/libbeat/
                - .buildkite/x-pack/pipeline.xpack.libbeat.yml
                - .buildkite/scripts
                - .buildkite/hooks
                #OSS
                - go.mod
                - pytest.ini
                - dev-tools/
                - testing/
                # x-pack
                - libbeat/

Copy link
Contributor Author

@sharbuz sharbuz Apr 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or you mean we shouldn't merge them?

Copy link
Contributor

@dliappis dliappis Apr 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they are the same, I am just saying that for consistency's sake we should have a section

                # x-pack
                - go.mod
                - pytest.ini
                - dev-tools/
                - libbeat/
                - testing/
                - x-pack/libbeat/

for all x-pack pipelines. The pipeline specific bits will, of course, remain above, so the total thing with my proposal would be:

- x-pack/<whatever beat>/ # exceptionally here, since it's x-pack/libbeat which is part of the the x-pack common section below we should skip this line
- .buildkite/x-pack/pipeline.xpack.libbeat.yml
- .buildkite/scripts/
- .buildkite/hooks/
# x-pack
- go.mod
- pytest.ini
- dev-tools/
- libbeat/
- testing/
- x-pack/libbeat/

config:
trigger: "beats-xpack-libbeat"
build:
commit: "${BUILDKITE_COMMIT}"
branch: "${BUILDKITE_BRANCH}"
env:
- BUILDKITE_PULL_REQUEST=${BUILDKITE_PULL_REQUEST}
- BUILDKITE_PULL_REQUEST_BASE_BRANCH=${BUILDKITE_PULL_REQUEST_BASE_BRANCH}
- GITHUB_PR_LABELS=${GITHUB_PR_LABELS}

- label: "Trigger x-pack/auditbeat"
Expand Down
18 changes: 1 addition & 17 deletions .buildkite/pull-requests.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,22 +79,6 @@
"skip_target_branches": [ ],
"skip_ci_on_only_changed": [ ],
"always_require_ci_on_changed": ["^x-pack/packetbeat/.*", "^.buildkite/.*", "^go.mod", "^pytest.ini", "^dev-tools/.*", "^libbeat/.*", "^testing/.*", "^x-pack/libbeat/.*"]
},
{
"enabled": true,
"pipelineSlug": "beats-xpack-libbeat",
"allow_org_users": true,
"allowed_repo_permissions": ["admin", "write"],
"allowed_list": [ ],
"set_commit_status": true,
"build_on_commit": true,
"build_on_comment": true,
"trigger_comment_regex": "^/test x-pack/libbeat$",
"always_trigger_comment_regex": "^/test x-pack/libbeat$",
"skip_ci_labels": [ ],
"skip_target_branches": [ ],
"skip_ci_on_only_changed": [ ],
"always_require_ci_on_changed": ["^x-pack/libbeat/.*", "^.buildkite/.*", "^go.mod", "^pytest.ini", "^dev-tools/.*", "^libbeat/.*", "^testing/.*", "^x-pack/libbeat/.*"]
}
]
}
}
25 changes: 9 additions & 16 deletions .buildkite/scripts/common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,14 @@ XPACK_MODULE_PATTERN="^x-pack\\/[a-z0-9]+beat\\/module\\/([^\\/]+)\\/.*"
# define if needed run the whole pipeline for the particular beat
[ -z "${run_filebeat+x}" ] && run_filebeat="$(buildkite-agent meta-data get run_filebeat --default "false")"
[ -z "${run_packetbeat+x}" ] && run_packetbeat="$(buildkite-agent meta-data get run_packetbeat --default "false")"
[ -z "${run_xpack_libbeat+x}" ] && run_xpack_libbeat="$(buildkite-agent meta-data get run_xpack_libbeat --default "false")"
[ -z "${run_xpack_metricbeat+x}" ] && run_xpack_metricbeat="$(buildkite-agent meta-data get run_xpack_metricbeat --default "false")"
[ -z "${run_xpack_packetbeat+x}" ] && run_xpack_packetbeat="$(buildkite-agent meta-data get run_xpack_packetbeat --default "false")"

# define if needed run ARM platform-specific tests for the particular beat
[ -z "${run_filebeat_arm_tests+x}" ] && run_filebeat_arm_tests="$(buildkite-agent meta-data get run_filebeat_arm_tests --default "false")"
[ -z "${run_packetbeat_arm_tests+x}" ] && run_packetbeat_arm_tests="$(buildkite-agent meta-data get run_packetbeat_arm_tests --default "false")"
[ -z "${run_xpack_libbeat_arm_tests+x}" ] && run_xpack_libbeat_arm_tests="$(buildkite-agent meta-data get run_xpack_libbeat_arm_tests --default "false")"
[ -z "${run_xpack_packetbeat_arm_tests+x}" ] && run_xpack_packetbeat_arm_tests="$(buildkite-agent meta-data get run_xpack_packetbeat_arm_tests --default "false")"

# define if needed run MacOS platform-specific tests for the particular beat
[ -z "${run_packetbeat_macos_tests+x}" ] && run_packetbeat_macos_tests="$(buildkite-agent meta-data get run_packetbeat_macos_tests --default "false")"
[ -z "${run_xpack_metricbeat_macos_tests+x}" ] && run_xpack_metricbeat_macos_tests="$(buildkite-agent meta-data get run_xpack_metricbeat_macos_tests --default "false")"
Expand All @@ -30,10 +30,6 @@ XPACK_MODULE_PATTERN="^x-pack\\/[a-z0-9]+beat\\/module\\/([^\\/]+)\\/.*"
# define if needed run cloud-specific tests for the particular beat
[ -z "${run_xpack_metricbeat_aws_tests+x}" ] && run_xpack_metricbeat_aws_tests="$(buildkite-agent meta-data get run_xpack_metricbeat_aws_tests --default "false")"

libbeat_changeset=(
"^libbeat/.*"
)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this on purpose? given that the PR is about x-pack/libbeat

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

libbeat pipeline has already merged.
Also, in this pipeline I've added one part from libbeat (we discussed that earlier):

beats/.buildkite/pipeline.yml

Line 268 in cd81faa

diff: "git diff --name-only origin/${GITHUB_PR_TARGET_BRANCH}...HEAD"

packetbeat_changeset=(
"^packetbeat/.*"
)
Expand All @@ -46,8 +42,12 @@ xpack_dockerlogbeat_changeset=(
"^x-pack/dockerlogbeat/.*"
)

xpack_libbeat_changeset=(
"^x-pack/libbeat/.*"
xpack_heartbeat_changeset=(
"^x-pack/heartbeat/.*"
)

xpack_filebeat_changeset=(
"^x-pack/filebeat/.*"
)

xpack_packetbeat_changeset=(
Expand Down Expand Up @@ -75,7 +75,6 @@ oss_changeset=(
)

xpack_changeset=(
"${xpack_libbeat_changeset[@]}"
"${oss_changeset[@]}"
)

Expand All @@ -90,18 +89,12 @@ packaging_changeset=(
)

case "${BUILDKITE_PIPELINE_SLUG}" in
"beats-libbeat")
BEAT_CHANGESET_REFERENCE=${libbeat_changeset[@]}
;;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we intentionally removing beats-libbeat? the PR is about x-pack/libbeat

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

answered here

"beats-packetbeat")
BEAT_CHANGESET_REFERENCE=${packetbeat_changeset[@]}
;;
"beats-winlogbeat")
BEAT_CHANGESET_REFERENCE=${winlogbeat_changeset[@]}
;;
"beats-xpack-libbeat")
BEAT_CHANGESET_REFERENCE=${xpack_libbeat_changeset[@]}
;;
"beats-xpack-metricbeat")
BEAT_CHANGESET_REFERENCE=${xpack_metricbeat_changeset[@]}
;;
Expand Down Expand Up @@ -347,7 +340,7 @@ are_conditions_met_mandatory_tests() {

are_conditions_met_arm_tests() {
if are_conditions_met_mandatory_tests; then #from https://github.com/elastic/beats/blob/c5e79a25d05d5bdfa9da4d187fe89523faa42afc/Jenkinsfile#L145-L171
if [[ "$BUILDKITE_PIPELINE_SLUG" == "beats-libbeat" || "$BUILDKITE_PIPELINE_SLUG" == "beats-packetbeat" ]]; then
if [[ "$BUILDKITE_PIPELINE_SLUG" == "beats-packetbeat" || "$BUILDKITE_PIPELINE_SLUG" == "beats-xpack-auditbeat" || "$BUILDKITE_PIPELINE_SLUG" == "beats-xpack-filebeat" ]]; then
if [[ "${GITHUB_PR_TRIGGER_COMMENT}" == "${BEATS_GH_ARM_COMMENT}" || "${GITHUB_PR_LABELS}" =~ ${BEATS_GH_ARM_LABEL} || "${!TRIGGER_SPECIFIC_ARM_TESTS}" == "true" ]]; then
return 0
fi
Expand Down
188 changes: 0 additions & 188 deletions .buildkite/scripts/generate_xpack_libbeat_pipeline.sh
View file
Open in desktop

This file was deleted.

Loading
Loading