Generation support for Microsoft Graph SDK #2721

manicminer · 2023-07-04T01:28:09Z

Depends on Importer bug fixes and improvements in #2917

This PR adds support for generating a Microsoft Graph SDK supporting the v1.0 and beta API versions.

SDK is located at github.com/hashicorp/go-azure-sdk/microsoft-graph
Uses the github.com/hashicorp/go-azure-sdk/sdk/client/msgraph base client
Common types, including models, constants, and predicates, are output to the microsoft-graph/common/{version}/models package
Services are structured like this:

Example Client

Example Get Operation

tombuildsstuff

Left a few comments inline but on the whole this is looking pretty good - nice work 👍

We do need to chat about the rollout of this, we're going to need to make some changes to hashicorp/go-azure-sdk and coordinate that rollout before this is merged (or rather, before Graph SDKs are generated) else we'll hit the Go module limit right away - but once that's done (and the comments are fixed up) - we should be good to get this in, so let's sync up on that soon?

tombuildsstuff · 2023-09-15T11:17:47Z

data/Pandora.Data/Helpers/Type.cs

@@ -1,6 +1,7 @@
 using System;
 using System.Collections.Generic;
 using System.Reflection;
+using System.Runtime.CompilerServices;


nit but this isn't technically needed?

Suggested change

using System.Runtime.CompilerServices;

tombuildsstuff · 2023-09-15T11:22:23Z

data/Pandora.Api/V1/Helpers/ObjectDefinition.cs

+    [JsonPropertyName("referenceIsCommonType")]
+    public bool? ReferenceIsCommonType { get; set; }


Rather than doing this against each API type, given v2 of the Data API exposes per-service level feature-toggles - perhaps we should a) expose and then b) lean on those toggles (e.g. via a new endpoint /v1/(resource-manager|microsoft-graph)/settings)?

Per my understanding at least CommonTypes are supposed to enabled/disabled at a Data Source (e.g. Resource Manager/Microsoft Graph) level - rather than toggled on a per-Service/API Version/Resource level?

tombuildsstuff · 2023-09-15T11:23:46Z

tools/generator-go-sdk/generator/common_types.go

+}
+
+func (s *CommonTypesGenerator) GenerateForSDK(input SDKInput) error {
+	input.VersionName = strings.ToLower(input.VersionName)


we should have a feature flag in here for UseCommonTypes? Presumably this can be fetched from the Data API (v2) since that has this toggle/information already - whilst that's not in v1 that would be pretty quick to add if needed

tombuildsstuff · 2023-09-15T11:26:49Z

tools/generator-go-sdk/generator/helpers_test.go

@@ -1,6 +1,7 @@
 package generator


fwiw the sdk package has a testhelpers package with a canonical reference for this now

tombuildsstuff · 2023-09-15T11:31:56Z

tools/generator-go-sdk/generator/stage_predicates.go

-		modelNames[*operation.ResponseObject.ReferenceName] = struct{}{}
+		modelName := *operation.ResponseObject.ReferenceName
+
+		if _, ok := data.models[modelName]; !ok {


presumably we could output the predicates into the common types directory instead?

tombuildsstuff · 2023-09-15T11:42:21Z

tools/generator-go-sdk/generator/templater_predicates.go

@@ -94,7 +94,7 @@ func (p predicateTemplater) templateForModel(predicateStructName string, name st
 		for _, fieldName := range fieldNames {
 			fieldVal := model.Fields[fieldName]

-			typeInfo, err := golangTypeNameForObjectDefinition(fieldVal.ObjectDefinition)
+			typeInfo, err := golangTypeNameForObjectDefinition(fieldVal.ObjectDefinition, nil) // TODO package path needed?


at the time of writing these are scoped to the package, but these might become common types (above) - so a TODO either way

tombuildsstuff · 2023-09-15T11:45:42Z

tools/generator-go-sdk/main.go

 	var serviceNames string

 	f := flag.NewFlagSet("generator-go-sdk", flag.ExitOnError)
 	f.StringVar(&input.apiServerEndpoint, "data-api", "http://localhost:5000", "-data-api=http://localhost:5000")
-	f.StringVar(&input.outputDirectory, "output-dir", "", "-output-dir=../generated-sdk-dev")
+	f.StringVar(&input.outputDirectoryPath, "output-dir", "", "-output-dir=../generated-sdk-dev")
+	f.StringVar(&input.sdkName, "sdk", "resource-manager", "-sdk=resource-manager")


fwiw we should look to update the automated tooling to specify this

pandora/tools/wrapper-automation/run.go

Lines 75 to 81 in 67702bd

args := []string{

fmt.Sprintf("-data-api=%s", dataApiUri),

}

if outputDirectory != "" {

args = append(args, fmt.Sprintf("-output-dir=%s", outputDirectory))

}

cmd := exec.Command("generator-go-sdk", args...)

and the scripts: https://github.com/hashicorp/pandora/tree/main/scripts

probably worth adding a resource manager command to the terraform generator at the same time?

pandora/tools/wrapper-automation/run.go

Lines 95 to 102 in 67702bd

args := []string{

"import",

fmt.Sprintf("-data-api=%s", dataApiUri),

}

if outputDirectory != "" {

args = append(args, fmt.Sprintf("-output-dir=%s", outputDirectory))

}

cmd := exec.Command("importer-rest-api-specs", args...)

tombuildsstuff · 2023-09-15T11:47:11Z

tools/generator-go-sdk/main.go

+			}
+			return &version, nil
+		},
+	},
 }

 func main() {


kinda outside the scope of this PR, but this is getting complex enough we should probably adopt mitchellh/cli here

tombuildsstuff · 2023-09-15T11:49:58Z

tools/sdk/resourcemanager/models.go

-	Type          ApiObjectDefinitionType `json:"type"`
+	NestedItem            *ApiObjectDefinition    `json:"nestedItem,omitempty"`
+	ReferenceName         *string                 `json:"referenceName,omitempty"`
+	ReferenceIsCommonType *bool                   `json:"referenceIsCommonType,omitempty"`


(as above) do we need this at this level? if we instead pulled the settings from the API we'd achieve the same thing, which might actually make both generators easier? e.g.

GET /services { "microsoft-graph": { "rootUri": "/v1/microsoft-graph", "usesCommonTypes": true } "resource-manager": { "rootUri": "/v1/resource-manager", "usesCommonTypes": false } }

Which then means the CLI itself could take an argument expecting the Data Source (e.g. Microsoft Graph/Resource Manager) pulled from the API? This has the benefit of features being in one place (but a short-time compile-time hit until we're on v2 of the Data API)

tombuildsstuff · 2023-09-15T11:51:14Z

tools/sdk/services/data.go

@@ -121,3 +121,16 @@ func GetResourceManagerServicesByName(client resourcemanager.Client, servicesToL
 		Services: resourceManagerServices,
 	}, nil
 }
+
+// GetResourceManagerCommonTypes returns the specified Services from the Data API
+func GetResourceManagerCommonTypes(client resourcemanager.Client) (*ResourceManagerCommonTypes, error) {


(fwiw once the v2 Data API is merged we'll update the SDK to remove the resourcemanager references, presumably having a data-api-sdk folder or similar?)

hc-github-team-tf-azure · 2023-09-19T01:30:14Z

No new resource IDs found.

tombuildsstuff · 2024-01-10T17:51:11Z

x-ref #3601

tombuildsstuff · 2024-02-20T10:24:26Z

I'm going to closing this PR in favour of #3754, which tracks finishing this support.

At this time the Source Data Type has been threaded through, however the Import Path and Common Types logic hasn't been ported across - but that's tracked in Stage 15 of #3754

manicminer changed the base branch from main to importer-msgraph July 4, 2023 01:29

manicminer force-pushed the importer-msgraph branch from 212f141 to eb8ba4c Compare July 8, 2023 23:47

manicminer force-pushed the generator-msgraph branch 2 times, most recently from 5cfa433 to 1534f87 Compare July 11, 2023 09:05

Base automatically changed from importer-msgraph to main August 2, 2023 21:02

manicminer force-pushed the generator-msgraph branch from 38712ff to 5ddb78c Compare August 2, 2023 21:28

manicminer added 2 commits September 8, 2023 01:56

generator-go-sdk: support for generating a Microsoft Graph SDK

5d17e00

generator-go-sdk: support for common types and predicates

9fee068

manicminer force-pushed the generator-msgraph branch from 5ddb78c to 9fee068 Compare September 8, 2023 01:02

manicminer changed the base branch from main to bugfix/msgraph-importer-id-detection September 8, 2023 01:02

manicminer force-pushed the generator-msgraph branch from 5884b91 to 9fee068 Compare September 8, 2023 01:03

manicminer marked this pull request as ready for review September 8, 2023 01:14

manicminer requested a review from tombuildsstuff September 8, 2023 01:14

hashicorp deleted a comment from hc-github-team-tf-azure Sep 8, 2023

manicminer added 3 commits September 8, 2023 02:39

generator-go-sdk: expand user-agent string, struct field documentation

609473d

data-api: support for flagging a reference as being a common type

39f3d0d

generator-go-sdk: minor tidying and refactoring

bca60a6

manicminer force-pushed the generator-msgraph branch from 8e21b32 to bca60a6 Compare September 8, 2023 09:28

generator-go-sdk: test fixes

d5aab5a

hashicorp deleted a comment from hc-github-team-tf-azure Sep 8, 2023

tombuildsstuff suggested changes Sep 15, 2023

View reviewed changes

tombuildsstuff added the waiting-response label Sep 15, 2023

Base automatically changed from bugfix/msgraph-importer-id-detection to main September 19, 2023 01:29

tombuildsstuff mentioned this pull request Feb 5, 2024

Meta: refactoring #3754

Open

tombuildsstuff closed this Feb 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generation support for Microsoft Graph SDK #2721

Generation support for Microsoft Graph SDK #2721

manicminer commented Jul 4, 2023 •

edited

Loading

tombuildsstuff left a comment

tombuildsstuff Sep 15, 2023

tombuildsstuff Sep 15, 2023

tombuildsstuff Sep 15, 2023

tombuildsstuff Sep 15, 2023

tombuildsstuff Sep 15, 2023

tombuildsstuff Sep 15, 2023

tombuildsstuff Sep 15, 2023

tombuildsstuff Sep 15, 2023

tombuildsstuff Sep 15, 2023

tombuildsstuff Sep 15, 2023

hc-github-team-tf-azure commented Sep 19, 2023

tombuildsstuff commented Jan 10, 2024

tombuildsstuff commented Feb 20, 2024

		[JsonPropertyName("referenceIsCommonType")]
		public bool? ReferenceIsCommonType { get; set; }

	args := []string{
	fmt.Sprintf("-data-api=%s", dataApiUri),
	}
	if outputDirectory != "" {
	args = append(args, fmt.Sprintf("-output-dir=%s", outputDirectory))
	}
	cmd := exec.Command("generator-go-sdk", args...)

	args := []string{
	"import",
	fmt.Sprintf("-data-api=%s", dataApiUri),
	}
	if outputDirectory != "" {
	args = append(args, fmt.Sprintf("-output-dir=%s", outputDirectory))
	}
	cmd := exec.Command("importer-rest-api-specs", args...)

Generation support for Microsoft Graph SDK #2721

Generation support for Microsoft Graph SDK #2721

Conversation

manicminer commented Jul 4, 2023 • edited Loading

Example Client

Example Get Operation

tombuildsstuff left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hc-github-team-tf-azure commented Sep 19, 2023

tombuildsstuff commented Jan 10, 2024

tombuildsstuff commented Feb 20, 2024

manicminer commented Jul 4, 2023 •

edited

Loading