Stability challenges and Insights for reg-pilot tests failures

[ronakseth96]

Observed discrepancies in the reg-pilot test outcomes across different environments. These discrepancies appear to be related to variations in OS, Node.js versions and required sleep intervals between operations, impacting test consistency and its successful run for rootsid_test preset.

code block, where the issuance process relies on credential retrieval and failures occur:

if (!credHolder) {
    await sendGrantMessage(issuerClient, issuerAID, recipientAID, cred);
    await sendAdmitMessage(recipientClient, recipientAID, issuerAID);

    credHolder = await retry(async () => {
        const cCred = await getReceivedCredential(recipientClient, cred.sad.d);
        assert(cCred !== undefined);
        return cCred;
    }, CRED_RETRY_DEFAULTS);
}

Insights:
During debugging, we observed that when credHolder is not populated, the retry logic initiates sequential calls to sendGrantMessage followed by sendAdmitMessage. This process continues until the credential is fully received, with the expectation that sendGrantMessage completes successfully before progressing to sendAdmitMessage. Most delays were concentrated in retries within the sendAdmitMessage phase, which required significantly more time to complete successfully compared to sendGrantMessage.

To improve test stability in regular runs, we experimented with incremental sleep intervals between calls. Adding sleep intervals before and after sendGrantMessage and following sendAdmitMessage yielded a few successful attempts.

• Aidar achieved stable test results with shorter sleep intervals, around sleep(2000) or sleep(3000).
• For me, the same configuration didn’t lead to consistent success, and longer sleep intervals were required. Through debugging, I increased the intervals and found that sleep(8000) before and after sendGrantMessage and sleep(16000) after sendAdmitMessage provided a few successful attempts.

We believe that the sleep duration requirements might vary due to the following environmental factors:
1. OS

• Aidar’s tests ran on Ubuntu, the same environment used in GitHub Actions, where shorter sleep intervals were successful.
• my tests ran on macOS, where significantly longer sleep intervals (e.g., sleep(8000) and sleep(16000)) were needed for tests to pass.

2. Node.js Version

• Aidar is having Node.js 20.18.0, while I am on version 20.12.0. This version difference could influence the timing or behavior of operations, impacting test stability.