Improve machine learning

[tms-epcc]

As explained in FY24 Q2 QU

Implementation of machine learning for active asteroids project (https://www.zooniverse.org/projects/orionnau/active-asteroids) . The Zooniverse has backend tools designed to support the use of machine learning (KADE:
https://github.com/zooniverse/kade/pkgs/container/kade BAJOR: https://github.com/zooniverse/bajor/pkgs/container/bajor). At present, they have been used for only one project, though they were designed to be reusuable. This project will increase the familiarity of the Rubin developer (Oyegoke) with these tools, assess their current status, and support an active Zooniverse project which is a Rubin precursor project.

[tms-epcc]

29/MAY/24

• A new task as agreed with the EPO team that takes priority over some of the previously identified tasks.
• Spec for this is being written

[tms-epcc]

From draft FY24 Annual Evaluation

The most substantial work this year has been the development and upgrading of the Zooniverse platform’s support for machine learning. While we anticipate that project teams will be responsible for the development of their own ML solutions, we wish to provide tools for those who want to incorporate ML more directly into their workflow. In particular, we anticipate users (i) using ML for advanced task assignment, for example by retiring subjects for which the machine has confident answers, or those where a small number of users agrees with the machine and (ii) developing full human in the loop systems where the machine is retrained as new data arrives. In the case of (ii) we anticipate relatively infrequent training, perhaps weekly.

Two modules, kade (https://github.com/zooniverse/kade/pkgs/container/kade) and bajor (https://github.com/zooniverse/bajor/pkgs/container/bajor) are provided to enable this work. Kade (the Knowledge and Discovery Engine) supports the use of Zooniverse data by ML systems, which are assumed to be running on the Zooniverse Azure cloud, and bajor is responsible scheduling Azure batch jobs for training machine learning systems. Both have been upgraded to newer versions of Rails, and documentation of both has been improved. As described in the work below, the next stage is to use these tools in production as a test of their capabilities. For a fuller description of their capabilities, we refer the reader to the repositories linked above.