Integrate SDK generation into build process

[slifty]

We've decided to have the service repository generate and publish sdk packages.

The best tool for this generation seems to be https://github.com/swagger-api/swagger-codegen, which is a java based tool. Although the generation and publication will be done via github CI we have a few additional considerations. Exploring those considerations is the first step!

[slifty]

Question 1: How to keep our version of swagger-codegen up to date

It would be Good for dependabot to manage the version of swagger-codegen for us / open PRs for new releases.

Dependabot supports Maven (pom.xml) and Gradle (build.gradle).

If we use maven then (and someone tell me if I'm wrong) it doesn't look like maven is intended to be used to download jars that are going to be used in external scripts; rather, the "Maven" way forward would be to write a maven plugin or java application that is responsible for building the sdk's, and so sdk generation would look something like maven clean compile.

There is a Maven plugin to this end, but unfortunately it only supports v2 of swagger-codegen (and therefore only v2 of the swagger spec, we are on v3).

I'm going to also look into whether or not we can just use Maven to cleanly download the jars specified as dependencies and then use those jars in a shell-based sdk build script.

If we don't want maven as a dev dependency and are comfortable being weird, I could write a script that pulls the latest version from pom.xml (so the version is managed by dependabot) and downloads a jar via curl (though we'd lose things like checksums / etc that we get for free from Maven)

---

UPDATE: sounds like gradle is what we want!

[slifty]

Worth noting: we need to be able to run build scripts locally.

This does mean that either (A) Java is going to become a development environment dependency for service or (B) Docker is going to become a development environment dependency for service.

I don't think we want Docker as the dependency though I'm open to being told that's not true!

So at the very least we'll need a JRE as a dev dependency. Stay tuned!

[slifty]

Gradle works just fine, but also it turns out I was wrong about the maven plugin only supporting v2 -- it just moved over here: https://mvnrepository.com/artifact/io.swagger.codegen.v3

Which is to say, we could:

1. Use the maven plugin, which means configuration would be in a pom.xml file and building would involve maven clean build
2. Use gradle to download the cli jar, and use a custom shell script to run the jar for the various languages.

I like the idea of leaning on the maven plugin (I'll see if there's a gradle equivalent too), rather than re-implementing similar things in bash.

[slifty]

The Maven approach is nice; the default output of the typescript compiler does not generate a package.json which means it isn't building into something that is publish-ready (I believe that pip / python is similarly lacking). It looks like this is just a configuration issue.

Interestingly it DOES generate a shell script for pushing the resulting code to a github repository! It seems maybe that pattern isn't so strange after all? Could be worth exploring.

[slifty]

I started digging in a bit to the actual generated code -- unfortunately it appears upstream's templates for typescript-fetch probably are not in the state we would want them to be. I raised a question with @kfogel in zulip around the higher-level question of upstream maintenance. I think it would be good to do in the long run; it isn't clear to me that upstream should be our first path to edits, however (as opposed to having our own forked version of the template that we can edit / improve without external negotiation).

Either way, the template for typescript does not feel usable out of the box. It has typos but also hasn't been edited in 3 years, relies on a 6-year-old dependency, and targets Node 8.

[kfogel]

@slifty Can we discuss this in the dev meeting tomorrow?

[jasonaowen]

@jvasile raised today that it would be helpful to have a script alongside the instructions.

[slifty]

We now have an initial version of the typescript template over in https://github.com/PhilanthropyDataCommons/sdk

My plan for this issue is to add a github action that will pull that project, invoke the generator, build the sdk, and publish it to npm. This is all possible, but there are two items I'm thinking through:

1. Thinking about SDK versioning (we have a swagger spec version AND a template version; either of those changing could be a reason for publishing a new package).
2. Trying to have the github action be smart enough to know when there's a new SDK version.

From an high level perspective the goal here is to generate and publish to npm every time there is a new specification version OR when there is a new template version.

The github actions in this repository are not going to be aware of new versions of the template.
The github actions in the template repository are not going to be aware of new versions of the API spec.

One solution would be to have a completely separate mechanism that watches both repositories for changes and invokes the build (e.g. via a cron).

Another solution would be to ONLY focus on CI / CD for situations where the api specification changes. When templates change we can manually release a new sdk version (or simply make it possible to manually trigger the publish github action). I think this is where I'd like to start.

---

I think the published NPM version should be the sum of the major / minor / patch of the template and the API specification. This might mean version numbers are skipped but it would enusre that they monotonically increase. This would also ensure that the nature of the change is aligned with semver (though in the event one package is v1 and the other is still v0 we might need to have some interesting logic to ensure the SDK is released as v0)