Stabilize SAML integration test cases for security dashboard CIs

[RyanL1997]

Description

Stabilize CI for front end.

Category

Test Enhancement

Issues Resolved

• Relate: Stabilize CI for dashboards repo #1624

Check List

• New functionality includes testing
• New functionality has been documented
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (0b72907) 66.29% compared to head (11d3b38) 66.29%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1641   +/-   ##
=======================================
  Coverage   66.29%   66.29%           
=======================================
  Files          93       93           
  Lines        2344     2344           
  Branches      317      317           
=======================================
  Hits         1554     1554           
  Misses        720      720           
  Partials       70       70           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[RyanL1997]

Results of 10 runs for each runner (Ubuntu + Windows):

Before fix

1. [The best rate] https://github.com/opensearch-project/security-dashboards-plugin/actions/runs/6788036743/job/18452161539?pr=1641
2. [The worst rate] https://github.com/opensearch-project/security-dashboards-plugin/actions/runs/6790899343/job/18461330944?pr=1641
3. Impacting sam's PR: https://github.com/opensearch-project/security-dashboards-plugin/actions/runs/6812401930/job/18524658952?pr=1502

After fix

1. https://github.com/opensearch-project/security-dashboards-plugin/actions/runs/6806732566/job/18508431615?pr=1641
2. https://github.com/opensearch-project/security-dashboards-plugin/actions/runs/6806554944/job/18507967620?pr=1641
3. https://github.com/opensearch-project/security-dashboards-plugin/actions/runs/6815132175/job/18533579549?pr=1641
4. https://github.com/opensearch-project/security-dashboards-plugin/actions/runs/6815132175/job/18533579549?pr=1641

[peternied]

Its good to see that these tests seem to be responsible for the issues with stability and these fixes look like they stop testing the scenario as written. I'm happy to mark these tests unreliable/ignored while we stabilize the CI.

As it is I don't think we should take this change.

[RyanL1997]

I have added some description of the reasons behind these changes, and the test results can be found in the above comment.

[RyanL1997]

Here is what I observed for http://localhost:5601/app/home vs http://localhost:5601/app/home#/:

1. First, after checking multiple versions (3.0.0, 2.11.0, 2.10.0 and 2.9.0), I think after we log into the dashboard by using the 'loginUrl' - http://localhost:5601/app/login, I'm sure the correct format of home page url should be http://localhost:5601/app/home#/.

2. By observing the browser while login, I noticed that the window url was http://localhost:5601/app/home during the loading of the dashboard screen and it will be http://localhost:5601/app/home#/ once everything is loaded:

a. full screen recording:

Screen.Recording.2023-11-09.at.3.03.57.PM.mov

b. url recording:

Screen.Recording.2023-11-09.at.3.05.06.PM.mov

3. The original test setup:

it('Login to Dashboards without nextUrl', async () => {
    const urlWithoutHash = `http://localhost:5601/app/home`;
    const loginUrl = `http://localhost:5601/app/login`;
    const driver = getDriver(browser, options).build();
    await driver.manage().deleteAllCookies();
    await driver.get(loginUrl);
    await driver.wait(until.elementsLocated(By.xpath(samlLogInButton)), 20000);
    await driver.findElement(By.xpath(samlLogInButton)).click();
    await driver.wait(until.elementsLocated(By.xpath(signInBtnXPath)), 20000);
    await driver.findElement(By.xpath(signInBtnXPath)).click();
    // TODO Use a better XPath.
    await driver.wait(
      until.elementsLocated(By.xpath('/html/body/div[1]/div/header/div/div[2]')),
      20000
    );
    const windowHash = await driver.getCurrentUrl();
    console.log('windowHash: ' + windowHash);
    expect(windowHash).toEqual(urlWithoutHash);
...

According to the above code and video, my hypothesis was that due to the loading delay for selenium driver + github runner, even with the locating of the header, line const windowHash = await driver.getCurrentUrl(); was getting the url during the loading of dashboard, which means it was getting the http://localhost:5601/app/home.

4. For proving my above hypothesis, I did an experiment:
   a. For commit-1, I re-enabled the 10 runs for each runner, and instead of using toContain(), I also switched the line back to expect(windowHash).toEqual(urlWithoutHash); And the CI passing result for ubuntu is 8/10 (link: https://github.com/opensearch-project/security-dashboards-plugin/actions/runs/6817812839/job/18542112378?pr=1641).
   b. For commit-2, in addition to the above commit, I hardcoded a sleep for 30 seconds, in between the login and get current window url. In another word, I gave the driver enough time to finish the loading of dashboard. And the CI passing result for ubuntu is 0/10 (link: https://github.com/opensearch-project/security-dashboards-plugin/actions/runs/6818153494/job/18543129417?pr=1641). We are getting this result, is because this time, by giving extra 30s, the dashboard finished loading completely, so that, according the videos showed above, the window url should be with #/ which fails the expectation of http://localhost:5601/app/home.

[RyanL1997]

Im looking into the unit test failure for now. But it seems like not relate to this changes.

[cwperks]

@RyanL1997 I raised a PR to address the babel imported: #1652