WF2 getting Lost when fetching a large patient list

[AishaHassen]

Describe the bug and expected behavior

• The MFS WF2 currently ends with a lost status when fetching a large number of patients(450+) from OMRS in its Get patient step.
• The Get patients step saves the patients in state and passes it down to the next job
• The next job doesn't even start because the state is too large to process
• See this run.

To Reproduce

1. Here is a link to a failed run on OpenFn.org which is indicative of the bug: link
2. To reproduce: run the workflow with a cursor date set as Jan 1, 2025

Possible solution

• The fetched patients from the get patients step get passed to downstream steps where they get mapped and reduced to a lighter list. The proposed solution, is to do the mapping of the Patients on the get patients step and reduce the size of the patient's array so that we don't bloat the state

expression.js

Link to the job itself in Github: https://github.com/OpenFn/msf-lime-mosul/blob/main/workflows/wf2/1-get-patients.js
Adaptor: http

To test/resolve

1. After the desired output is working locally (from the CLI), please open a pull request
2. Please test the change on OpenFn.org by re-running this run (link) and confirming that the work order doesn't end in a Lost state

Toggl

MSF-2024

[aleksa-krolls]

thanks for this @AishaHassen ... in which step does the mapping happen currently? Are we talking about moving the mapping from one step to another?

[AishaHassen]

Currently the mapping is happening on step Upsert TEIs and we want to move it up to Get Patients maybe even rename get and map patients and the actual Upsert operation on DHIS2 can happen in the Upsert TEIs step.

[aleksa-krolls]

@mtuchi I'd welcome your estimate here so we can think about cost implications of moving forward with this. But then let's you/me/ @AishaHassen discuss after stand-up and decide if it's worth the investment once we get updates from product team.

[mtuchi]

Implementation Notes

• Identify all jobs that uses patient data
• Identify all jobs that uses mappings
• Update each job that needs mapping to fetch the record from collection and remove the mapping from state
• Move patient mappings from Upsert teis step to Get patient step
• Update mapping in other jobs as well.
• Test the whole workflow

[mtuchi]

Pushing the product team to solve the lost run issue on lightning

[aleksa-krolls]

Closing this issue, because we decided that small enhancements here might not be enough to address the lost runs issue, and that there's probably a bigger platform-level fix needed. We can always reopen this issue if we change our minds.

Joe is working on a fix to Lost runs (see slack) that I think will really address this problem.