Backport 1.7 to 1.2 (#420)

* Adding support for structured dataset (#369) Signed-off-by: pmahindrakar-oss <[email protected]> * added dynamic_job_spec_uri to dynamic workflow metadata and node execution closure (#360) Signed-off-by: Daniel Rammer <[email protected]> * Use TokenCache in ClientCredentialsTokenSourceProvider (#377) * Init customTokenSource.refreshTime (#381) Signed-off-by: Andrew Dye <[email protected]> * added DataLoadingConfig to K8sPod message (#368) Signed-off-by: Daniel Rammer <[email protected]> * Add Reasons field to TaskExecutionClosure to track time-series of reasons (#382) * added a time-series of reasons to the TaskExecution closure Signed-off-by: Daniel Rammer <[email protected]> * added docs Signed-off-by: Daniel Rammer <[email protected]> * actually finishing docs too Signed-off-by: Daniel Rammer <[email protected]> --------- Signed-off-by: Daniel Rammer <[email protected]> * Create service for runtime metrics (#367) * added span messages Signed-off-by: Daniel Rammer <[email protected]> * added endpoints to service Signed-off-by: Daniel Rammer <[email protected]> * generated mocks Signed-off-by: Daniel Rammer <[email protected]> * removed get task execution metrics rpc Signed-off-by: Daniel Rammer <[email protected]> * added EXECUTION_IDLE category Signed-off-by: Daniel Rammer <[email protected]> * updated PLUGIN_EXECUTION to PLUGIN_RUNTIME Signed-off-by: Daniel Rammer <[email protected]> * removed recorded_at on workflow and node level events Signed-off-by: Daniel Rammer <[email protected]> * added docs for task event reported_at field Signed-off-by: Daniel Rammer <[email protected]> * removed GetNodeExecutionMetrics endpoint - will implement later if necessary Signed-off-by: Daniel Rammer <[email protected]> * updated docs Signed-off-by: Daniel Rammer <[email protected]> * added reported_at for node execution events Signed-off-by: Daniel Rammer <[email protected]> * fixed typo Signed-off-by: Daniel Rammer <[email protected]> * fixed typos and removed dead code Signed-off-by: Daniel Rammer <[email protected]> * updated categories Signed-off-by: Daniel Rammer <[email protected]> * added workflow setup and teardown categories Signed-off-by: Daniel Rammer <[email protected]> * simplified span message and moved to flyteidl.core Signed-off-by: Daniel Rammer <[email protected]> --------- Signed-off-by: Daniel Rammer <[email protected]> * Remove misleading token refresh logic from client credentials token source provider (#383) * Out of core plugin (#378) * Add backend plugin system service Signed-off-by: Kevin Su <[email protected]> * Add backend plugin system service Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> * update state Signed-off-by: Kevin Su <[email protected]> * update state Signed-off-by: Kevin Su <[email protected]> * dics Signed-off-by: Kevin Su <[email protected]> * Remove output prefix from get request Signed-off-by: Kevin Su <[email protected]> * update Signed-off-by: Kevin Su <[email protected]> * remove prev state Signed-off-by: Kevin Su <[email protected]> * update proto Signed-off-by: Kevin Su <[email protected]> * remove error message Signed-off-by: Kevin Su <[email protected]> * update comment Signed-off-by: Kevin Su <[email protected]> * make generate Signed-off-by: Kevin Su <[email protected]> * Rename the service Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Feat: Add `ElasticConfig` message type for torch elastic training (#394) * Add elastic config args to pytorch proto Signed-off-by: Fabio Graetz <[email protected]> * Add elastic config message type for torchrun training Signed-off-by: Fabio Graetz <[email protected]> --------- Signed-off-by: Fabio Graetz <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Co-authored-by: Ketan Umare <[email protected]> * Retract 1.4.x (#397) Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> * Data addresses #minor (#391) Signed-off-by: Yee Hing Tong <[email protected]> * Refactor kf-operator plugins configs and support setting different specs for different replica groups (#386) * refactor kubeflow operators proto Signed-off-by: Yubo Wang <[email protected]> * add back the original proto for backward compatible Signed-off-by: Yubo Wang <[email protected]> * clean up comments Signed-off-by: Yubo Wang <[email protected]> * add kubeflow.rs Signed-off-by: Yubo Wang <[email protected]> * add elastic config Signed-off-by: Yubo Wang <[email protected]> * add command to MPI Signed-off-by: Yubo Wang <[email protected]> * add slots and command to mpi spec Signed-off-by: Yubo Wang <[email protected]> --------- Signed-off-by: Yubo Wang <[email protected]> Co-authored-by: Yubo Wang <[email protected]> * add user_identifier (#388) Signed-off-by: byhsu <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Co-authored-by: byhsu <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> * Add envs to execution spec (#400) Signed-off-by: Kevin Su <[email protected]> * Support union and none type in flyteidl (#401) * add support for Union Scalar Signed-off-by: Yubo Wang <[email protected]> * support union type and literals Signed-off-by: Yubo Wang <[email protected]> * change union type extraction Signed-off-by: Yubo Wang <[email protected]> --------- Signed-off-by: Yubo Wang <[email protected]> Co-authored-by: Yubo Wang <[email protected]> Co-authored-by: Kevin Su <[email protected]> * Rename user_identity to execution_identity (#402) Signed-off-by: byhsu <[email protected]> Co-authored-by: byhsu <[email protected]> * Single literal in GetDataResponse (#404) Signed-off-by: Yee Hing Tong <[email protected]> * Add namespace to execution system metadata (#406) Signed-off-by: Katrina Rogan <[email protected]> * Add oauth2 http proxy client (#405) Signed-off-by: byhsu <[email protected]> * Rename externalPluginService to AgentService (#410) * Rename externalPluginService to AgentService Signed-off-by: Kevin Su <[email protected]> * nit Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Add external_plugin_service proto back to the idl (#413) * Add external-plugin-service proto back to the idl Signed-off-by: Kevin Su <[email protected]> * update idl Signed-off-by: Kevin Su <[email protected]> * update idll Signed-off-by: Kevin Su <[email protected]> * update idll Signed-off-by: Kevin Su <[email protected]> * AsyncAgentService Signed-off-by: Kevin Su <[email protected]> --------- Signed-off-by: Kevin Su <[email protected]> * Rerun make generate Signed-off-by: eduardo apolinario <[email protected]> --------- Signed-off-by: pmahindrakar-oss <[email protected]> Signed-off-by: Daniel Rammer <[email protected]> Signed-off-by: Andrew Dye <[email protected]> Signed-off-by: Kevin Su <[email protected]> Signed-off-by: Fabio Graetz <[email protected]> Signed-off-by: eduardo apolinario <[email protected]> Signed-off-by: Yee Hing Tong <[email protected]> Signed-off-by: Yubo Wang <[email protected]> Signed-off-by: byhsu <[email protected]> Signed-off-by: Katrina Rogan <[email protected]> Co-authored-by: pmahindrakar-oss <[email protected]> Co-authored-by: Dan Rammer <[email protected]> Co-authored-by: Andrew Dye <[email protected]> Co-authored-by: Kevin Su <[email protected]> Co-authored-by: Fabio M. Graetz, Ph.D <[email protected]> Co-authored-by: Fabio Grätz <[email protected]> Co-authored-by: Ketan Umare <[email protected]> Co-authored-by: eduardo apolinario <[email protected]> Co-authored-by: Yee Hing Tong <[email protected]> Co-authored-by: Yubo Wang <[email protected]> Co-authored-by: Yubo Wang <[email protected]> Co-authored-by: ByronHsu <[email protected]> Co-authored-by: byhsu <[email protected]> Co-authored-by: Katrina Rogan <[email protected]>
flyteorg · Jun 28, 2023 · 635f2e6 · 635f2e6
1 parent 61820a4
commit 635f2e6
Show file tree

Hide file tree

Showing 61 changed files with 17,194 additions and 530 deletions.
diff --git a/clients/go/admin/auth_interceptor.go b/clients/go/admin/auth_interceptor.go
@@ -3,10 +3,12 @@ package admin
 import (
 	"context"
 	"fmt"
+	"net/http"
 
 	"github.com/flyteorg/flyteidl/clients/go/admin/cache"
 	"github.com/flyteorg/flyteidl/gen/pb-go/flyteidl/service"
 	"github.com/flyteorg/flytestdlib/logger"
+	"golang.org/x/oauth2"
 
 	"google.golang.org/grpc/codes"
 	"google.golang.org/grpc/status"
@@ -50,6 +52,21 @@ func shouldAttemptToAuthenticate(errorCode codes.Code) bool {
 	return errorCode == codes.Unauthenticated
 }
 
+// Set up http client used in oauth2
+func setHTTPClientContext(ctx context.Context, cfg *Config) context.Context {
+	httpClient := &http.Client{}
+
+	if len(cfg.HTTPProxyURL.String()) > 0 {
+		// create a transport that uses the proxy
+		transport := &http.Transport{
+			Proxy: http.ProxyURL(&cfg.HTTPProxyURL.URL),
+		}
+		httpClient.Transport = transport
+	}
+
+	return context.WithValue(ctx, oauth2.HTTPClient, httpClient)
+}
+
 // NewAuthInterceptor creates a new grpc.UnaryClientInterceptor that forwards the grpc call and inspects the error.
 // It will first invoke the grpc pipeline (to proceed with the request) with no modifications. It's expected for the grpc
 // pipeline to already have a grpc.WithPerRPCCredentials() DialOption. If the perRPCCredentials has already been initialized,
@@ -62,6 +79,8 @@ func shouldAttemptToAuthenticate(errorCode codes.Code) bool {
 // be able to find and acquire a valid AccessToken to annotate the request with.
 func NewAuthInterceptor(cfg *Config, tokenCache cache.TokenCache, credentialsFuture *PerRPCCredentialsFuture) grpc.UnaryClientInterceptor {
 	return func(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
+		ctx = setHTTPClientContext(ctx, cfg)
+
 		err := invoker(ctx, method, req, reply, cc, opts...)
 		if err != nil {
 			logger.Debugf(ctx, "Request failed due to [%v]. If it's an unauthenticated error, we will attempt to establish an authenticated context.", err)

diff --git a/clients/go/admin/config.go b/clients/go/admin/config.go
@@ -79,6 +79,9 @@ type Config struct {
 	// find the full schema here https://github.com/grpc/grpc-proto/blob/master/grpc/service_config/service_config.proto#L625
 	// Note that required packages may need to be preloaded to support certain service config. For example "google.golang.org/grpc/balancer/roundrobin" should be preloaded to have round-robin policy supported.
 	DefaultServiceConfig string `json:"defaultServiceConfig" pdflag:",Set the default service config for the admin gRPC client"`
+
+	// HTTPProxyURL allows operators to access external OAuth2 servers using an external HTTP Proxy
+	HTTPProxyURL config.URL `json:"httpProxyURL" pflag:",OPTIONAL: HTTP Proxy to be used for OAuth requests."`
 }
 
 var (

diff --git a/clients/go/admin/config_flags.go b/clients/go/admin/config_flags.go
diff --git a/clients/go/admin/config_flags_test.go b/clients/go/admin/config_flags_test.go
diff --git a/clients/go/admin/mocks/AgentServiceClient.go b/clients/go/admin/mocks/AgentServiceClient.go
diff --git a/clients/go/admin/mocks/AgentServiceServer.go b/clients/go/admin/mocks/AgentServiceServer.go