Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test sharding #2810

Open
wants to merge 5 commits into
base: vvm/refactor_sdkv2_detailed_diff_tests
Choose a base branch
from
+7,188 −154
Open

Test sharding #2810

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e020e7f
test sharding
VenelinMartinov Jan 6, 2025
14a4d52
disable go setup caching on windows
VenelinMartinov Jan 7, 2025
6c6e852
move to 15 shards
VenelinMartinov Jan 7, 2025
0ac8012
bump
VenelinMartinov Jan 7, 2025
138e1d0
Revert irrelevant changes
VenelinMartinov Jan 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 73 additions & 3 deletions .github/workflows/build-and-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ on:
required: false
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
TOTAL_SHARDS: 15

jobs:
test:
Expand All @@ -21,6 +22,68 @@ jobs:
go-version: [1.22.x, 1.23.x]
platform: [ubuntu-latest, macos-latest, windows-latest]
feature-flags: ["DEFAULT", "PULUMI_TF_BRIDGE_ACCURATE_BRIDGE_PREVIEW"]
shard: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
exclude:
- platform: windows-latest
feature-flags: "PULUMI_TF_BRIDGE_ACCURATE_BRIDGE_PREVIEW"
- platform: macos-latest
feature-flags: "PULUMI_TF_BRIDGE_ACCURATE_BRIDGE_PREVIEW"
- platform: windows-latest
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of defining shard as [0, 1, 2, 3, ...], then excluding each option we don't want, can we define shard as [0] and then add explicit includes for other shard options. That will reduce the length of the shard list by half.

shard: 1
Comment on lines +31 to +32
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the idea that every OS runs shard 0, and only ubuntu runs the rest of the shards? If so, that needs a comment explaining so.

Also, why don't we shard all OSs?

- platform: windows-latest
shard: 2
- platform: windows-latest
shard: 3
- platform: windows-latest
shard: 4
- platform: windows-latest
shard: 5
- platform: windows-latest
shard: 6
- platform: windows-latest
shard: 7
- platform: windows-latest
shard: 8
- platform: windows-latest
shard: 9
- platform: windows-latest
shard: 10
- platform: windows-latest
shard: 11
- platform: windows-latest
shard: 12
- platform: windows-latest
shard: 13
- platform: windows-latest
shard: 14
- platform: macos-latest
shard: 1
- platform: macos-latest
shard: 2
- platform: macos-latest
shard: 3
- platform: macos-latest
shard: 4
- platform: macos-latest
shard: 5
- platform: macos-latest
shard: 6
- platform: macos-latest
shard: 7
- platform: macos-latest
shard: 8
- platform: macos-latest
shard: 9
- platform: macos-latest
shard: 10
- platform: macos-latest
shard: 11
- platform: macos-latest
shard: 12
- platform: macos-latest
shard: 13
- platform: macos-latest
shard: 14
runs-on: ${{ matrix.platform }}
steps:
- name: Install pulumi
Expand All @@ -35,18 +98,25 @@ jobs:
go-version: ${{ matrix.go-version }}
cache-dependency-path: |
**/go.sum
# disable caching on windows because it's very slow
# see https://github.com/actions/setup-go/issues/495
cache: ${{ matrix.platform != 'windows-latest' }}
- name: export feature flags
run: echo ${{ matrix.feature-flags }}=true >> $GITHUB_ENV
if: ${{ matrix.platform != 'windows-latest' && matrix.feature-flags != 'DEFAULT' }}
- name: export feature flags
run: echo ${{ matrix.feature-flags }}=true >> $env:GITHUB_ENV
if: ${{ matrix.platform == 'windows-latest' && matrix.feature-flags != 'DEFAULT' }}
- name: Build
run: make build
- name: Build PF
run: cd pkg/pf && make build
- name: Shard tests
run: echo "SHARD_CMD=$(go run github.com/VenelinMartinov/sharder@9d09afeb1c053b4a0263fbaa5c3c9da1ca2d10e7 --total ${{ env.TOTAL_SHARDS }} --index ${{ matrix.shard }} --output testoutput.txt --format make)" >> $GITHUB_ENV
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is github.com/VenelinMartinov/sharder different from github.com/pulumi/shard? Why can't we use github.com/pulumi/shard?

if: ${{ matrix.platform == 'ubuntu-latest' }}
- name: Test
run: make VERBOSE=true test_shard
if: ${{ matrix.platform == 'ubuntu-latest' }}
- name: Test
run: make test
if: ${{ matrix.platform != 'ubuntu-latest' }}
- name: Upload coverage reports to Codecov
# If we have a CODECOV_TOKEN secret, then we upload it to get a coverage report.
#
Expand Down
4 changes: 3 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,6 @@ testing/coverage.txt
pf/coverage.out
pf/tests/coverage.out

pulumi-hcl-lint
pulumi-hcl-lint

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we adding this. We should fix the test that erroneously generates this file instead of ignoring it.

schema-tpsdkv2.json
14 changes: 14 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,20 @@ test:: install_plugins
PULUMI_TERRAFORM_BRIDGE_TEST_PROVIDER=$(shell pwd)/bin/pulumi-terraform-bridge-test-provider \
go test -count=1 -coverprofile="coverage.txt" -coverpkg=./... -timeout 2h -parallel ${TESTPARALLELISM} ./...

test_shard:: install_plugins
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also needs a comment explaining what it does, and that it is provided for CI.

@mkdir -p bin
go build -o bin ./internal/testing/pulumi-terraform-bridge-test-provider
PULUMI_TERRAFORM_BRIDGE_TEST_PROVIDER=$(shell pwd)/bin/pulumi-terraform-bridge-test-provider \
go test -count=1 -coverprofile="coverage.txt" -coverpkg=./... -timeout 2h -parallel ${TESTPARALLELISM} ./... ${SHARD_CMD}

test_profile:: install_plugins
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a comment explaining what it does.

@mkdir -p bin
go build -o bin ./internal/testing/pulumi-terraform-bridge-test-provider
rm -f testoutput.txt
PULUMI_TERRAFORM_BRIDGE_TEST_PROVIDER=$(shell pwd)/bin/pulumi-terraform-bridge-test-provider \
go test -count=1 -timeout 2h -parallel ${TESTPARALLELISM} ./... -json | \
jq -c -r 'select((.Action == "pass" or .Action == "fail") and .Test != null)' | tee testoutput.txt

# Run tests while accepting current output as expected output "golden"
# tests. In case where system behavior changes intentionally this can
# be useful to run to review the differences with git diff.
Expand Down
150 changes: 0 additions & 150 deletions schema-tpsdkv2.json
View file
Open in desktop

This file was deleted.

Loading
Loading