Ability to select SDK languages #1063

ringods · 2024-08-28T13:11:37Z

Update Sep 9: dropped the enableAutoRelease from this PR. Will redo this as a separate PR.

Round 2 to add support for third-party providers:

Fix: Pass the configured organization to 2 more places where it is needed
languages (default: all supported languages): third party providers can configure a subset of the supported languages if they don't have the publishing infrastructure ready for all the languages. Changes:
- changes the build matrix to only the set of configured languages
- will install only the tools needed for the given value of the language matrix. Changed this because installing the Java tools failed when no Java SDK is available.

The changes in this PR are focussed on the main workflow. Testing this on pulumiverse/pulumi-acme already got me this far:

https://github.com/pulumiverse/pulumi-acme/actions/runs/10596199445

danielrbradley

Overall approach looks good. Need clarification on if the comment-dispatch change was accidental or if we need to adjust the settings name somehow.

Out of interest - why can't external providers run the auto-release hooks? Is it possible just to make this work for them too?

danielrbradley · 2024-08-28T13:44:05Z

provider-ci/internal/pkg/templates/provider/.github/workflows/command-dispatch.yml

@@ -1,3 +1,4 @@
+#{{- if .Config.enableAutoRelease -}}#


This seems a little odd - this workflow is about auto-commenting, not auto-releasing.

@ringods what was your thoughts here?

I believe this workflow is also pretty pulumi-specific and outside parties may want to enable and disable this, but the naming of the flag makes this confusing.

provider-ci/internal/pkg/templates/bridged-provider/.github/workflows/build_sdk.yml

ringods · 2024-08-28T14:48:44Z

@danielrbradley AFAICS, the auto release setup depends on the Pulumi bot setup. If I want this to work for third party providers, I want to find a solution based on the standard github permissions: setting within a workflow. Can't solve everything at once unfortunately, so I chose to comment out more.

guineveresaenger

I think with the enableAutoRelease flag it might make sense to add an external provider to the test-providers so we can see one in action?

provider-ci/internal/pkg/templates/bridged-provider.config.yaml

provider-ci/internal/pkg/templates/bridged-provider/.github/workflows/build_sdk.yml

guineveresaenger · 2024-08-29T15:59:07Z

provider-ci/internal/pkg/templates/provider/.github/workflows/command-dispatch.yml

@@ -1,3 +1,4 @@
+#{{- if .Config.enableAutoRelease -}}#


I believe this workflow is also pretty pulumi-specific and outside parties may want to enable and disable this, but the naming of the flag makes this confusing.

guineveresaenger · 2024-08-29T16:09:26Z

provider-ci/internal/pkg/templates/bridged-provider.config.yaml

@@ -75,6 +75,9 @@ freeDiskSpaceBeforeTest: false
 # Set to true to clear disk space before running sdk build jobs.
 freeDiskSpaceBeforeSdkBuild: false

+# Enables the automatic release of the provider
+enableAutoRelease: true


I love this flag; I think the terminology is a bit confusing. When I first read it, my thought was, "wait, what is auto release?"
I think your PR comment makes sesne as a code comment here.

enableAutoRelease (default: true): used for Pulumi, but typically not for third party providers. Changes:
- Disables the publish step of the main workflow
- Prevents generating repository dispatch workflows like command-dispatch and release-command

Further thoughts:
Could we call it "enablePublish" or even "skipPublish"? Would it make sense to have a global skipWorkflow flag to allow for maximum configurability?
We could even have a skipPulumiInternals switch since this is explicitly targeted at external users?

Earlier today, I had a short review call for this PR with @danielrbradley. To prevent adding a lot more configuration options to the setup, an alternative setup would be to create a separate template folder aimed at thirdparty bridged providers, reshuffle some files between the different folders and update generate.go here:

ci-mgmt/provider-ci/internal/pkg/generate.go

Lines 69 to 75 in b6d3a9f

switch templateName {

case "bridged-provider":

// Render more specific templates last to allow them to override more general templates.

return []string{"provider", "dev-container", "bridged-provider"}, nil

default:

return nil, fmt.Errorf("unknown template: %s", templateName)

}

How does that sound to you?

Changing `node` to `nodejs` in tool installation task This brings consistency with the name of the language used in the GHA `language` build matrix

danielrbradley

Happy with all these changes 👍

Follow-up to #1063 The nuget command fails if dotnet wasn't installed.

Follow-up to #1063 The nuget command fails if dotnet wasn't installed. Example failure: https://github.com/pulumi/pulumi-azure/actions/runs/10815039270/job/30003728480?pr=2387 This only affects providers which have enabled `freeDiskSpaceBeforeTest` because they will have a default version of dotnet available, and therefore also wasn't caught by the downstream tests. For the sake of consistency, we'll also only configure the pip virtualenv if we're atually running python tests.

This change pulls the latest ci-mgmt changes in order to reduce the size of provider binaries. In detail, we're now setting the `-s -w` linker flags to strip debug symbols. This cuts the provider binary size by roughly 300MB, resulting in a 32% smaller provider binary. Note: I also had to change `toolVersions.node` to `toolVersions.nodejs` because the configuration is now type checked. The `toolVersions.node` configuration option was renamed to `nodejs` in this PR: pulumi/ci-mgmt#1063

ringods added 2 commits August 28, 2024 13:25

Fix code block annotation

b9ae5cb

Pass the configured organization name

06d7162

ringods requested review from danielrbradley, guineveresaenger and iwahbe August 28, 2024 13:11

danielrbradley reviewed Aug 28, 2024

View reviewed changes

guineveresaenger reviewed Aug 29, 2024

View reviewed changes

ringods force-pushed the disable-autorelease branch from 649b65f to 9f270e3 Compare September 9, 2024 11:14

ringods changed the title ~~Disable autorelease & ability to select SDK languages~~ Ability to select SDK languages Sep 9, 2024

Ability to configure the set of language SDKs

02bf75d

Changing `node` to `nodejs` in tool installation task This brings consistency with the name of the language used in the GHA `language` build matrix

ringods force-pushed the disable-autorelease branch from 9f270e3 to 02bf75d Compare September 9, 2024 11:20

danielrbradley approved these changes Sep 9, 2024

View reviewed changes

Use languages in (pre)release.

36c1eb1

ringods requested a review from guineveresaenger September 9, 2024 16:09

guineveresaenger approved these changes Sep 10, 2024

View reviewed changes

ringods added this pull request to the merge queue Sep 10, 2024

Merged via the queue into master with commit b7adcae Sep 10, 2024
6 checks passed

ringods deleted the disable-autorelease branch September 10, 2024 09:23

danielrbradley added a commit that referenced this pull request Sep 11, 2024

Only add nuget source for dotnet tests

9b44cf1

Follow-up to #1063 The nuget command fails if dotnet wasn't installed.

danielrbradley mentioned this pull request Sep 11, 2024

Only add nuget source for dotnet tests #1079

Merged

flostadler mentioned this pull request Nov 18, 2024

Update ci-mgmt to strip debug symbols from provider binaries pulumi/pulumi-aws#4786

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ability to select SDK languages #1063

Ability to select SDK languages #1063

ringods commented Aug 28, 2024 •

edited

Loading

danielrbradley left a comment

danielrbradley Aug 28, 2024

danielrbradley Aug 29, 2024

guineveresaenger Aug 29, 2024

ringods commented Aug 28, 2024

guineveresaenger left a comment

guineveresaenger Aug 29, 2024

guineveresaenger Aug 29, 2024

ringods Aug 29, 2024

danielrbradley left a comment

	switch templateName {
	case "bridged-provider":
	// Render more specific templates last to allow them to override more general templates.
	return []string{"provider", "dev-container", "bridged-provider"}, nil
	default:
	return nil, fmt.Errorf("unknown template: %s", templateName)
	}

Ability to select SDK languages #1063

Ability to select SDK languages #1063

Conversation

ringods commented Aug 28, 2024 • edited Loading

danielrbradley left a comment

Choose a reason for hiding this comment

danielrbradley Aug 28, 2024

Choose a reason for hiding this comment

danielrbradley Aug 29, 2024

Choose a reason for hiding this comment

guineveresaenger Aug 29, 2024

Choose a reason for hiding this comment

ringods commented Aug 28, 2024

guineveresaenger left a comment

Choose a reason for hiding this comment

guineveresaenger Aug 29, 2024

Choose a reason for hiding this comment

guineveresaenger Aug 29, 2024

Choose a reason for hiding this comment

ringods Aug 29, 2024

Choose a reason for hiding this comment

danielrbradley left a comment

Choose a reason for hiding this comment

ringods commented Aug 28, 2024 •

edited

Loading