API for Workflows #1887
Comments
taylordowns2000
changed the title
|
@josephjclark added you to collaborate with @jyeshe |
|
@taylordowns2000 for the update on a first draft/version, if there is any Presence on the workflow I am going to reject the update from the API okay? It's the simplest and safest validation for now and would give us time to think about prioritization rules involving the API. |
|
The Workflow structure used by the API - the JSON body to GET and POST - needs to be consistent with other representations within Lightning. To that end, I recommend we use the same I think that means the There's also the question of options: we recognise a bunch of options on the run which I think are saved to the workflow? I suppose we could exclude options for now and just use default values Quick note that the CLI and Runtime use a different JSON workflow representation internally. That structure is a bit easier to write out by hand, meaning that WCS for now will have to write out a slightly more verbose JSON object to create their new workflows. This can be improved later when we update the openfn adaptor (and if we add CLI support (although this is what deploy does so I don't know if we need more) we can simplify the structure there) |
|
Is it also worth noting that in the coming weeks we hope to get work
started in earnest on a React front-end for the canvas and inspector which
would communicate with the Phoenix app using this same workflow JSON api?
…On Mon, Dec 16, 2024 at 5:51 AM Joe Clark ***@***.***> wrote:
The Workflow structure used by the API - the JSON body to GET and POST -
needs to be consistent with other representations within Lightning.
To that end, I recommend we use the same { triggers, jobs, edges, options
} format used by the workflow diagram and the Runs that are compiled for
the worker (obviously some of the run-specific properties, like
dataclip_id, would be excluded).
I think that means the workflows schema in workflow.ex is the correct
structure. Minus runs and snapshots.
There's also the question of options: we recognise a bunch of options
<https://github.com/OpenFn/kit/blob/5b3e3e9e1a4f7b573cf152cedcfe251c7d592a73/packages/lexicon/lightning.d.ts#L45>
on the run which I think are saved to the workflow?
type LightningPlanOptions = {
run_timeout_ms?: number;
sanitize?: SanitizePolicies;
start?: StepId;
output_dataclips?: boolean;
run_memory_limit_mb?: number;
payload_limit_mb?: number;
};
I suppose we could exclude options for now and just use default values
—
Reply to this email directly, view it on GitHub
<#1887 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACCUBLJTCDSF6X6RY4BKQI32F2WCHAVCNFSM6AAAAABTR635TOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNBVGI2DKNZZG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
Taylor Downs
CEO, OpenFn <https://www.openfn.org>
|
|
@taylordowns2000 Yes, worth bearing in mind. For today we're doing a simple bulk upsert, where you POST the whole workflow and it'll all be saved. The workflow editor will want to PUT or PATCH a partial diff, probably compliant with JSON patch to maintain parity with the current editor. I suggest we do this as a second pass in a couple of weeks. |
|
Roger and I have agreed on this: This explicitly surfaces the data model - in Roger's words, "the project id is king". We do not want an API which works like "we'll return all workflows that you're allowed to see with your credential), because it's arbitrary and PAT tokens aren't bound to a project or anything. So instead, we require that a project id must be included in the request (even if it's technically redundant) POST and PUT must include a whole workflow object. The project id in the body must match the id in the path. We'll probably add a PATCH later which accepts a diff in the body (probably JSON Patch). You can easily imagine a |
|
Personally I would go with the proposed REST api having the On the other hand, this way having the Joe (@josephjclark) here is a draft with the GET 1, GET many and POST for creation in case it helps anyway on the client side: #2792 |
|
@josephjclark I was thinking about the external applications API usage and to enable their systems to be fully automated, without manual input for the projects in this case, currently they would need a GET /workflows regardless of the project. May we go with the REST version making the Sorry about it but another detail that passed through my eyes I am noticing now is that considering a desired replacement for the existing jobs API it behaves exactly like this. There is an optional |
|
Refinement of this after a call with Taylor. The project id should be at the base of the path always. That fixes everything and ensures the project id is still king. The workflow API will be: We could maybe implement "get projects" down at the root Later we might add other APIs along these lines: |
|
One question coming out of the PR is: what to do about IDs for new steps and edges in the body? The strict solution is for Lightning to demand that all ids in the body are UUIDs, putting the emphasis on the client to generate them. I am uncomfortable with this, simply because for many clients it's very hard to do. I also think it's a bit of an artificial problem and pretty easy for the server to solve. In our WCS use-case for example, we don't provide a good mechanism to job code right now to generate a UUID. Like it's literally impossible. We'll need to extend the adaptor for API support. And even then, creating workflows with UUIDs is kinda hard because you need to not just generate an id for each step, but you need to cross-reference it elsewhere in the workflow. More generally, the way the CLI works is that it's not very fussy about defining ids. It's quite happy for me to run a workflow with an id like I appreciate that this is a technical and low-level API, but it is still a human facing one. If I wanted to support a user-friendly API to allow third-parties to create Workflows, I wouldn't be strict with them about ID formats. I'd ask that IDs be internally consistent but arbitrary, and I would swap them out before loading them into Lightning's database. So I'd really like to ask for a rule that says: when creating new steps or edges, the id you submit, and any references to it, will be replaced by a UUID by the server. Existing steps or edges must use the server's UUID (the mapping is not persisted). An updated workflow will be returned to you (an id map would be more efficient in the case of large workflows, maybe we should just do that). Maybe I should speak to someone in more detail tomorrow about WCS and make sure I'm super sure of the requirements for that specific workflow. |
|
I've gotten hold of the WCS workflow and I have a few observations:
I still think the server should handle id replacement. But we could also solve this in the |
|
TBH I really don't think server-side id replacement is a big deal and we just should push on and do it. It could be implemented for about the same effort that I've spent arguing for it 🤷 |
|
This issue is moved to blocked pending discussion with @stuartc on Monday to discuss:
|
|
Tracking documentation effort by @josephjclark for the Workflows API: Input on docs requirements from the implementation team, we're keeping documentation minimal. Further iteration on the documentation can be scheduled on need basis |
|
And for a full API spec, it can be found here: @stuartc this might need to be publicly available via endpoint or ideally by a hosted API spec UI. I'd suggest postman or readme.com. |
WCS has a use case where they want to create and update workflows(with steps) from a step. This was available in V1, and it blocks migration to v2 for this edge case.
A typical use case can be to build a new workflow every time a new form is created in CommCare.
We need to expose an API that developers can call from OpenFn or an external application to:
All endpoints should be authenticated with the user’s personal access token(Whatever is best, technically.)We also need to add documentation on how to use the Workflow JSON API.
No changes to the UI are required here.
Two important things to note:
lib/lightning_web/controllers/api/job_controller.ex) and runs (lib/lightning_web/controllers/api/run_controller.ex) is now outdated. Please consider how the new workflows REST api may (??) replace the outdatedjobs_controllerand how it does or does not relate to the upcoming workorders/runs API for GovStack. Update RunController and JSON api for GovStack compliance #1656The text was updated successfully, but these errors were encountered: