ML Pipeline v2

[mihow]

Summary

Bringing our ML pipeline out of beta.

List of Major Changes

• New Processing Services section. Allows users to add an endpoint themselves and serve machine learning pipelines from their own compute infrastructure or local machine. When a Service is added, the pipelines it has available are registered as options that the project's users can choose for processing images.

• Begin storing label lists / category maps that the algorithms use. This model is called AlgorithmCategoryMap and maps the class index from the last model layer to the actual categories they represent. The categories have both a simple text label "Species name" as well as an object of metadata with an optional GBIF key, taxon rank, etc. These are used to create taxon entries in the Antenna database as well as show the top 3,5,10, N number of predictions from a model rather than just the top 1 which is saved as the occurrence's determination.

• Moves job logs to their own field on the job model to reduce DB writes and overwriting the status field when writing logs.

• Job status and progress fields have been fixed so that the status & progress (# images done) are not toggling between states, or ending on a strange value.

• Update what is considered a job failure: If more than 50% of images in a job fail, or if any result saving fails.

• Add "task_type" field to algorithms so we can determine if its a classification model vs. a detection model and so on.

• Handle results from multiple algorithms. The best score from all (non-intermediate) algorithms is used to determine the species determination of an occurrence (not just the latest prediction). Identifiers can now see and Agree with the result from any classifier, including an intermediate classifier like Moth/Non-Moth.

• Faster & more stable saving of results. Results from the ML backend are now saved more efficiently after each batch of images and error handling is greatly improved.

Related Issues

Closes #607
Closes #310

Unblocks several tickets coming up. Including:

• Using the Moth/non-moth intermediate model as a solo / terminal classifier Make a pipeline for only the binary classifier #621
• Allow reprocessing same detections with new classifier (without running detector again)
• Can apply post-processing filters of species by region (filter and re-weight the scores)
• Can do genus & higher taxon roll-ups
  Can use confidence score algorithms other than softmax on the Antenna side (e.g. temperature calibrated)
• Can better send current results back to backend to determine reprocessing needs
• Ready for detection tracking across frames!

Screenshots

Deployment Notes

Needs to be deployed with corresponding ML backend changes here:
RolnickLab/ami-data-companion#67

Checklist

• I have tested these changes appropriately.
• I have added and/or modified relevant tests.
• I updated relevant documentation or comments.
• I have verified that this PR follows the project's coding standards.
• Any dependent changes have already been merged to main.

[netlify]

✅ Deploy Preview for ami-dev canceled.

Name	Link
🔨 Latest commit	59b4867
🔍 Latest deploy log	https://app.netlify.com/sites/ami-dev/deploys/67964e1675c7480008670562

[sentry-io]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ ConnectionError: Error while reading from redis:6379 : (104, 'Connection reset by peer') ami.jobs.tasks.run_job View Issue
• ‼️ OperationalError: the connection is closed ami.jobs.tasks.run_job View Issue
• ‼️ ConnectionError: Error while reading from redis:6379 : (104, 'Connection reset by peer') ami.jobs.tasks.run_job View Issue
• ‼️ OperationalError: the connection is closed ami.jobs.tasks.run_job View Issue

_{Did you find this useful? React with a 👍 or 👎}