Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci(actions): trigger E2E tests on CircleCI manually from GitHub Actions #8426

Closed
wants to merge 2 commits into from
Closed

ci(actions): trigger E2E tests on CircleCI manually from GitHub Actions #8426

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4aa90ce
ci(actions): generate e2e tests matrix
jijiechen Nov 21, 2023
520f15d
ci(actions): trigger E2E tests on CircilCI manually from GitHub Actions
jijiechen Nov 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,27 @@ parameters:
ubuntu_image:
type: string
default: "ubuntu-2204:2022.10.2"
manual_e2e:
type: boolean
default: false
e2e_param_k8s_version:
type: string
default: ""
e2e_param_arch:
type: string
default: ""
e2e_param_parallelism:
type: integer
default: 1
e2e_param_target:
type: string
default: ""
e2e_param_cni_network_plugin:
type: string
default: flannel
e2e_param_legacy_kds:
type: boolean
default: false
# See https://circleci.com/docs/2.0/configuration-reference/#commands-requires-version-21.
commands:
install_build_tools:
Expand Down Expand Up @@ -517,3 +538,20 @@ workflows:
requires: [build]
- distributions:
requires: [test, container-structure]
manual-e2e:
when:
equal: [true, << pipeline.parameters.manual_e2e >>]
jobs:
- go_cache:
name: go_cache-<< pipeline.parameters.e2e_param_arch >>
arch: << pipeline.parameters.e2e_param_arch >>
- build:
name: build
jijiechen marked this conversation as resolved.
Show resolved Hide resolved
- e2e:
k8sVersion: << pipeline.parameters.e2e_param_k8s_version >>
parallelism: << pipeline.parameters.e2e_param_parallelism >>
target: << pipeline.parameters.e2e_param_target >>
arch: << pipeline.parameters.e2e_param_arch >>
cniNetworkPlugin: << pipeline.parameters.e2e_param_cni_network_plugin >>
legacyKDS: << pipeline.parameters.e2e_param_legacy_kds >>
requires: [build, go_cache-<< pipeline.parameters.e2e_param_arch >>]
77 changes: 77 additions & 0 deletions .github/actions/e2e/action.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
name: 'Kuma E2E'
description: 'Run end to end tests for kuma'
inputs:
k8sVersion:
description: version of k3s to use or "kind" and "kindIpv6"
default: v1.28.1-k3s1
required: false
parallelism:
description: level of parallelization
default: '1'
required: false
target:
description: makefile target without e2e prefix
default: ""
required: false
arch:
description: The golang arch
default: amd64
required: false
cniNetworkPlugin:
description: The CNI networking plugin to use [flannel | calico]
default: flannel
required: false
legacyKDS:
description: if should run tests with new implementation of KDS
default: 'false'
required: false
secureCircleCIToken:
description: The personal access token used to invoke CircleCI API (not project API Token)
default: ''
required: false
runs:
using: "composite"
steps:
- run: |
echo "All inputs:"
echo "Running with: \
k8s: ${{ inputs.k8sVersion }} \
target: ${{ inputs.target }} \
parallelism: ${{ inputs.parallelism }} \
arch: ${{ inputs.arch }} \
cniNetworkPlugin: ${{ inputs.cniNetworkPlugin }} \
"
shell: bash
- run: |
# Trigger CircleCI manually, reference: https://github.com/CircleCI-Public/trigger-circleci-pipeline-action/blob/main/dist/index.js#L16310

BODY='{"parameters": { "manual_e2e": true, "e2e_param_k8s_version": "${{ inputs.k8sVersion }}", "e2e_param_arch": "${{ inputs.arch }}", "e2e_param_parallelism": ${{ inputs.parallelism }}, "e2e_param_target": "${{ inputs.target }}", "e2e_param_cni_network_plugin": "${{ inputs.cniNetworkPlugin }}", "e2e_param_legacy_kds": ${{ inputs.legacyKDS }} } }'

if [ "${{ github.ref_type }}" == "tag" ]]; then
BODY=$(echo $BODY | jq -rc '.+= {"tag": "${{ github.ref_name }}"}')
else
BODY=$(echo $BODY | jq -rc '.+= {"branch": "${{ github.ref_name }}"}')
fi

CIRCLE_CI_API_PATH=https://circleci.com/api/v2/project/gh/${{ github.repository }}/pipeline
echo "Calling CircleCI api with parameters:
URL: $CIRCLE_CI_API_PATH
BODY: $BODY"

if [ "${{ inputs.secureCircleCIToken }}" == "" ]; then
echo "Skipping request CircleCI because secret 'CIRCLECI_TOKEN' not set."
exit 0
fi

VERBOSE=
if [ "${{ runner.debug }}" == "1" ]; then
VERBOSE='-v'
fi

curl $VERBOSE --fail -X POST $CIRCLE_CI_API_PATH \
--header "content-type: application/json" \
--header "x-attribution-login: ${{ github.actor }}" \
--header "x-attribution-actor-id: ${{ github.actor }}" \
--header "Circle-Token: ${{ inputs.secureCircleCIToken }}" \
--data "$BODY"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason we don't use: https://github.com/CircleCI-Public/trigger-circleci-pipeline-action is that we can't pass params? I guess the problem with this is that we don't block until completion no?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe instead of having parameters to the circleci job we should just read https://github.com/CircleCI-Public/trigger-circleci-pipeline-action/tree/main#gha_meta in the circleCI job ?s

Copy link
Member Author

@jijiechen jijiechen Nov 22, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, we can't pass custom parameters using that action. That's why I call it manually. The GHA_Meta provides a way to pass custom parameters, we have to composite all params into one in this way, and we don't have a easy way to extract these params as part of the CircleCI workflow (using their setup job could be a way, that might require significant changes to the whole strcture).

In terms of blocking until completion, I believe that action does not provide this ability either. I read through the source code, and checkout of their readme, they are calling the same API with the one in my code and they only provide state of the just-created pipeline as an output as well.

For blocking until complete, we may laverage their other APIs to all workflows and periodically check latest state. Doc at https://circleci.com/docs/api/v2/index.html#operation/getWorkflowById

shell: bash
163 changes: 163 additions & 0 deletions .github/workflows/build-test-distribute.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ on:
tags: ["*"]
pull_request_target:
branches: ["master", "release-*"]
env:
K8S_MIN_VERSION: v1.23.17-k3s1
K8S_MAX_VERSION: v1.28.1-k3s1
jobs:
check:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -170,3 +173,163 @@ jobs:
trap on_exit EXIT
make docker/push
make docker/manifest
gen_e2e_matrix:
runs-on: ubuntu-latest
needs: ["build"]
outputs:
slow: ${{ steps.generate-matrix.outputs.slow }}
main: ${{ steps.generate-matrix.outputs.main }}
legacy_kds: ${{ steps.generate-matrix.outputs.legacy_kds }}
cni: ${{ steps.generate-matrix.outputs.cni }}
steps:
- name: Install jq
run: |
sudo apt-get install -y jq
- id: generate-matrix
name: Generate matrix
run: |
SKIP_CI_LABEL="${{ contains(github.event.pull_request.labels.*.name, 'ci/skip-test') || contains(github.event.pull_request.labels.*.name, 'ci/skip-e2e-test') }}"
if [[ "$SKIP_CI_LABEL" == "true" ]]; then
echo "Skipping e2e jobs because the pull request contained a 'ci/skip-*' label."
exit 0
fi

ALL_K8S_VERSION='{"k8sVersion":["kind", "kindIpv6", "${{ env.K8S_MIN_VERSION }}", "${{ env.K8S_MAX_VERSION }}"]}'
ALL_ARCH='{"arch": ["amd64", "arm64"]}'
ALL_TARGET='{"target": ["kubernetes", "universal", "multizone"]}'
SINGLE_ENV='{"k8sVersion": ["${{ env.K8S_MAX_VERSION }}"], "target": ["multizone"], "arch":["amd64"]}'
ALL_LEGACY_KDS='{"legacyKDS": [true]}'
ALL_CNI='{"cniNetworkPlugin": ["calico"]}'

# arm64 is treated as a non-priority
EXCLUDE_ARM64='[{"arch": "arm64"}]'

# kind should only be used when testing ipv6 or with e2e-universal
EXCLUDE_KIND='[{"k8sVersion":"kind", "target": "kubernetes"}, {"k8sVersion":"kind", "target": "multizone"}]'

# universal only runs on kind
EXCLUDE_UNIVERSAL='[{"target":"universal", "k8sVersion":"${{ env.K8S_MIN_VERSION }}"}, {"target":"universal", "k8sVersion":"${{ env.K8S_MAX_VERSION }}"}]'

# non priority main e2e
EXCLUDE_MAIN_NON_PRIORITY='[{"k8sVersion":"kindIpv6"}, {"k8sVersion": "${{ env.K8S_MIN_VERSION }}"}]'

# Default matrix
SLOW=$(echo '{}' | jq -rc ". += $ALL_K8S_VERSION | . += $ALL_ARCH")
MAIN=$(echo '{}' | jq -rc ". += $ALL_K8S_VERSION | . += $ALL_ARCH | . += $ALL_TARGET" | jq -rc ".exclude += $EXCLUDE_KIND | .exclude += $EXCLUDE_UNIVERSAL" )
LEGACY_KDS=$(echo '{}' | jq -rc ". += $ALL_LEGACY_KDS | . += $SINGLE_ENV")
CNI=$(echo '{}' | jq -rc ". += $ALL_CNI | . += $SINGLE_ENV")

RUN_FULL_MATRIX="${{ contains(github.event.pull_request.labels.*.name, 'ci/run-full-matrix') }}"
if [[ "$RUN_FULL_MATRIX" == "false" ]]; then
echo "Skipping non priority e2e tests, because no label 'ci/run-full-matrix' was found on the pull request."

SLOW='' # target==""
LEGACY_KDS='' # legacyKDS=="true"
CNI='' # cniNetworkPlugin=="calico"
MAIN=$(echo $MAIN | jq -rc ".exclude += $EXCLUDE_ARM64 | .exclude += $EXCLUDE_MAIN_NON_PRIORITY")
fi

echo "slow=$SLOW" >> $GITHUB_OUTPUT
echo "main=$MAIN" >> $GITHUB_OUTPUT
echo "legacy_kds=$LEGACY_KDS" >> $GITHUB_OUTPUT
echo "cni=$CNI" >> $GITHUB_OUTPUT
e2e_slow:
name: "legacy-k8s:${{ matrix.arch }}-${{ matrix.k8sVersion }}"
runs-on: ubuntu-latest
needs: ["gen_e2e_matrix"]
if: needs.gen_e2e_matrix.outputs.slow != ''
strategy:
matrix: ${{ fromJSON(needs.gen_e2e_matrix.outputs.slow) }}
steps:
- run: |
echo "k8sVersion: ${{ matrix.k8sVersion }}"
echo "arch: ${{ matrix.arch }}"
- name: Install jq
run: |
sudo apt-get install -y jq
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: ./.github/actions/e2e
with:
k8sVersion: ${{ matrix.k8sVersion }}
arch: ${{ matrix.arch }}
parallelism: 3
target: ""
secureCircleCIToken: ${{ secrets.CIRCLECI_TOKEN }}
e2e_main:
name: "${{ matrix.target }}:${{ matrix.arch }}-${{ matrix.k8sVersion }}"
runs-on: ubuntu-latest
needs: ["gen_e2e_matrix"]
if: needs.gen_e2e_matrix.outputs.main != ''
strategy:
matrix: ${{ fromJSON(needs.gen_e2e_matrix.outputs.main) }}
steps:
- run: |
echo "k8sVersion: ${{ matrix.k8sVersion }}"
echo "target: ${{ matrix.target }}"
echo "arch: ${{ matrix.arch }}"
- name: Install jq
run: |
sudo apt-get install -y jq
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: ./.github/actions/e2e
with:
k8sVersion: ${{ matrix.k8sVersion }}
target: ${{ matrix.target }}
arch: ${{ matrix.arch }}
secureCircleCIToken: ${{ secrets.CIRCLECI_TOKEN }}
e2e_legacy_kds:
name: "${{ matrix.target }}:${{ matrix.arch }}-${{ matrix.k8sVersion }}-legacy-kds"
runs-on: ubuntu-latest
needs: ["gen_e2e_matrix"]
if: needs.gen_e2e_matrix.outputs.legacy_kds != ''
strategy:
matrix: ${{ fromJSON(needs.gen_e2e_matrix.outputs.legacy_kds) }}
steps:
- run: |
echo "k8sVersion: ${{ matrix.k8sVersion }}"
echo "target: ${{ matrix.target }}"
echo "arch: ${{ matrix.arch }}"
echo "legacyKDS: ${{ matrix.arch }}"
- name: Install jq
run: |
sudo apt-get install -y jq
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: ./.github/actions/e2e
with:
k8sVersion: ${{ matrix.k8sVersion }}
target: ${{ matrix.arch }}
arch: ${{ matrix.arch }}
legacyKDS: ${{ matrix.legacyKDS }}
secureCircleCIToken: ${{ secrets.CIRCLECI_TOKEN }}
e2e_calico:
name: "${{ matrix.target }}:${{ matrix.arch }}-${{ matrix.k8sVersion }}-calico"
runs-on: ubuntu-latest
needs: ["gen_e2e_matrix"]
if: needs.gen_e2e_matrix.outputs.cni != ''
strategy:
matrix: ${{ fromJSON(needs.gen_e2e_matrix.outputs.cni) }}
steps:
- run: |
echo "k8sVersion: ${{ matrix.k8sVersion }}"
echo "target: ${{ matrix.target }}"
echo "arch: ${{ matrix.arch }}"
echo "cniNetworkPlugin: ${{ matrix.cniNetworkPlugin }}"
- name: Install jq
run: |
sudo apt-get install -y jq
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: ./.github/actions/e2e
with:
k8sVersion: ${{ matrix.k8sVersion }}
target: ${{ matrix.arch }}
arch: ${{ matrix.arch }}
cniNetworkPlugin: ${{ matrix.cniNetworkPlugin }}
secureCircleCIToken: ${{ secrets.CIRCLECI_TOKEN }}