Better error handling and status tracking for GitHub actions

[christad92]

Our current design of the GitHub sync is such that when a user triggers the sync there is no way for them to know if it is successful. When this is successful, it is great but when there is an error and the sync is not successful, it can be catastrophic.

When a sync operation fails, we should trigger the failure alert as well so that they receive an email that it failed. In the future, we should allow them to see a list of GitHub syncs on their projects and the status of those operations.

Notify users when GitHub sync failure

• OpenFn -> GitHub
  • Alert the user who initiated the sync about the failure
• GitHub -> OpenFn
  • Alert the users on the project that a sync from GitHub to OpenFn has failed. This should be treated as a failure alert and ONLY users that enabled failure alerts should be notified.

Show users the last time a project synced on the project list page

• We are adding a new field named Latest sync to the projects table (in the db) and it should store the last time a project was synced to GitHub.
• Add a Latest Sync date field to the project list table to show the last time the project was synced. (In the future we want to allow users to sync their projects wherever they like – bitbucket, GitLab, S3 bucket, or any other storage service)
  Please do not remove the last updated field.

[taylordowns2000]

@stuartc , @midigofrank , this needs to be spiked

[midigofrank]

Hmm, I think we will need more than just latest_sync on projects table. I think we will need to track syncs separately on a different table, maybe github_syncs that can keep track of:

• project
• snapshots/workflows created
• user
• github workflow run

We can use webhooks to know when the github workflow run finishes. https://docs.github.com/en/webhooks/webhook-events-and-payloads#workflow_run

[christad92]

@theroinaochieng why is this marked as blocked?

I have created an issue for showing and alerting users when Sync breaks here: #2965

Happy to chat about this.

[christad92]

cc; @midigofrank please can you share why this is blocked?

[midigofrank]

My bad @christad92, I thought I already added it here.

In order to use these webhooks, we need a way to identify the github workflow run and be able to match it with a user in lightning.

We can use webhooks to know when the github workflow run finishes. https://docs.github.com/en/webhooks/webhook-events-and-payloads#workflow_run

The problem is that when initiating a sync, we don't get anything back that is related to the github workflow run. The current implementation creates a workflow dispatch event https://docs.github.com/en/rest/actions/workflows?apiVersion=2022-11-28#create-a-workflow-dispatch-event which has no return body.

This issue is blocked because there is no clear way to go around this. We should set aside some time to do an analysis of what is possible/not possible