Add workflows API to create, get, update and list

[jyeshe]

Description

This PR adds an API that allows to create, update, get and list workflows. The authentication uses a Bearer token and the user of the Personal Access Token must belong to the project related to the workflow.

For POST and PUT the body should be a JSON encoded workflow having {name, project_id, edges, jobs, triggers}.
A PATCH can be also used to update a workflow passing only a difference of the fields that needs to be changed.

When certain workflow preconditions are not met, it returns 422 (Unprocessable Entity) with an error message describing the violation:

• "The triggers cannot be replaced, only edited or added." (cannot delete a webhook trigger for example)
• "A workflow can have only one trigger enabled at a time."

A POST or PUT or PATCH might also fail if the outcome would be having more active workflows than the limit (when running on the billing app). For this case it also returns a 422 with the message:

"Your plan has reached the limit of active workflows."

Some additional information on protocol syntax in regards to paths and responses:

• All paths follow a nested REST resource on projects (e.g PATCH /api/projects/:project_id/workflows/:id).
• Getting a single workflow returns a JSON with a workflow
• Getting a list of workflows returns a JSON with a workflows
• All other paths return a JSON {"id": <workflow_uuid>, "error": <error_message>}

Closes #1887

Validation steps

Create a Personal Access Token (PAT) on Lightning user profile menu and make these curl requests with authorization: Bearer <the_PAT>, accept: application/jsonand content-type: application/json

Example of POST to create using $token and project_id 86a7888e-8a07-4673-9dbf-43ba016dc790

curl -s -X POST -H "content-type: application/json" -H "authorization: Bearer $token" localhost:4000/api/projects/86a7888e-8a07-4673-9dbf-43ba016dc790/workflows -d "{\"id\":null,\"name\":\"work1\",\"project_id\":\"86a7888e-8a07-4673-9dbf-43ba016dc790\",\"edges\":[{\"enabled\":true,\"id\":\"a8f136af-4527-4009-9122-f6e67fdf0a44\",\"source_trigger_id\":\"858f21e6-f301-4c56-8bcf-66b7378d1e54\",\"target_job_id\":\"66810fb2-e770-4d71-9c3c-44a0354f650d\",\"condition_type\":\"always\",\"source_job_id\":null}],\"jobs\":[{\"id\":\"66810fb2-e770-4d71-9c3c-44a0354f650d\",\"body\":\"fn(state => { return {...state, extra: \\\"data\\\"} })\",\"name\":\"job-9\",\"adaptor\":\"@openfn/language-common@latest\",\"inserted_at\":null,\"updated_at\":null}],\"triggers\":[{\"id\":\"858f21e6-f301-4c56-8bcf-66b7378d1e54\",\"comment\":null,\"custom_path\":null,\"cron_expression\":null,\"type\":\"webhook\",\"enabled\":true}],\"inserted_at\":null,\"updated_at\":null}"
{"error":null,"id":"2eed727b-99a1-4bf8-8312-1ab460f4fd47"}

it shall return a JSON in this format: {"error":null,"id":"2eed727b-99a1-4bf8-8312-1ab460f4fd47"}

After that execute curl commands for:
GET /api/projects/:projectId/workflows
GET /api/projects/:projectId/workflows/:workflowId
PUT /api/projects/:projectId/workflows/:workflowId
PATCH /api/projects/:projectId/workflows/:workflowId

Additional notes for the reviewer

Currently PUT and PATCH are handled by the same update function so can be used interchangeably until a final validation is added.

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

• Code generation (copilot but not intellisense)
• Learning or fact checking
• Strategy / design
• Optimisation / refactoring
• Translation / spellchecking / doc gen
• Other
• I have not used AI

You can read more details in our Responsible AI Policy

Pre-submission checklist

• I have performed a self-review of my code.
• I have implemented and tested all related authorization policies. (e.g., :owner, :admin, :editor, :viewer)
• I have updated the changelog.
• I have ticked a box in "AI usage" in this PR

[jyeshe]

This is where I'm at now:
⛔ get workflows with invalid project id (expect 404 and error object, got 400 webpage)

Sorry Joe, previously it was implemented only on the get workflow, now it's done for the index

⛔ get workflows for project that does not exist (expect 404, got 401)

The reason it's returning 401 Unauthorized it's because the project id is used for ACL. The user could check the project id on 401 but if 404 is really helpful for the user we can add a query or subquery for all operations. In any case I've added the 401 on the spec with a description "User not allowed to access the project" for existing projects.

⛔ PUT a new job with arbitrary ids

The arbitrary ids are supported only on the POST because the triggers cannot be replaced. If arbitrary ids are allowed only for jobs we could have a single rule for POST and PUT. However the PUT we need to be aware that all jobs would be replaced and I need to check what is behaviour of the edges on this jobs full update.

⛔ PUT a new orphaned job without an id

On PUT everything is expected to have an id for being an update. It's doable however.

[jyeshe]

Take aways from our meeting:

• There is no particular rule for disconnected/orphan jobs in regards to ids. Client apps need to pass ids on all jobs.
• All sucessfull operations will return the resulting workflow.
• Make sure that PUT is able to remove jobs and edges.
• RESTful PATCH shall be available (put back/reenable).

[jyeshe]

@jyeshe any idea why the tests are timing out 🤔 ?

to be continued here: #2822

[midigofrank]

Hey @jyeshe , this looks really great.

There's a comment I've left regarding using if..else instead of cond.
Also, there's a conflict on the CHANGELOG.

[midigofrank]

Shouldn't we just use if..else here?

[stuartc]

Great work @jyeshe, left some comments; I think at least we need to add the other condition_... fields to the Edge @derive property.

[stuartc]

I think we also need condition_expression and condition_label to allow for JS based conditions no?

[jyeshe]

Done

[stuartc]

Is there not a way to make the distinction between emails (in this case) and other uniqueness violations?

[jyeshe]

It's possible but we gonna need to call a different translation. Opened an issue for identifying and changing on all places of the app to have a specific message for email: #2824