(feature) Support rerun mode

[BorisOsipov]

See docs here https://github.com/reportportal/documentation/blob/master/src/md/src/DevGuides/rerun.md

[AutomationReddy]

@BorisOsipov Any update on this?

[BorisOsipov]

@AutomationReddy nope.

[ghost]

@BorisOsipov is there any progress on this item?
I'm looking into using specFileRetries WDIO config option with ReportPortal. At the moment wdio-reportportal-reporter creates a new test suite item for each spec file retry. And that doesn't work well with RP, since it supports retries only on a test item level. And for suite-level retry it needs a rerun, which in turn needs a new launch.

The only way I can see this working right now is by creating a new RP launch for each spec file retry while keeping the original launch started. Then for each spec file retry one needs to create a new launch with retry_of flag set. Here are the detailed steps:

• create a new launch for whole wdio run
• test fails and spec file is retried: create a new launch with rerun_of set to main launch
  • test succeeds: finish the rerun launch
  • test fails and spec file is retried more than once: create a new launch with rerun_of set to previous rerun launch
  • if test fails and no more spec retries left: finish the rerun launch
• wdio run is complete, finish main launch

This seems a cumbersome solution, and it also needs some sort of a flag from WDIO in onSuiteStart indicating the retry mode and number of reties left. I cannot see anything like this coming from WDIO.

Any thoughts on this?

[BorisOsipov]

is there any progress on this item?

nope. I don't read rerun docs mindfully..

I'm looking into using specFileRetries

this issue is about support rerun in common e.g. when you run yarn run wdio few times and want one launch at the end. But it is ok we can discuss specFileRetries here.

specFileRetries from WDIO and RP rerun\retry logic is big pain. In general as I see your algorithm is correct and should work.

This seems a cumbersome solution,

yes it is. do you really need it? maybe some services like https://webdriver.io/docs/wdio-rerun-service/ can help?

it also needs some sort of a flag from WDIO in onSuiteStart indicating the retry mode and number of reties left

Yes it is. Also we have to save somewhere launchIds for all launches and correct set what launch we rerun on each retry.

[ghost]

@BorisOsipov wdio-rerun-service doesn't really work in my case. I don't believe there is a way to integrate it with RP.
But on the other hand the idea of having a dedicated service for RP reruns sounds interesting. Might consider building it. Thanks for bringing up this idea!

[AdamOakman]

@BorisOsipov @hs-abarchuk did you guys find any way/workaround to make specFileRetries option work correctly with report portal? I was able to configure report portal with the project that uses gherkin retries quite nicely but I fail to do the same for the project that uses spec files with specFileRetries option without duplicating items within the launch

[BorisOsipov]

@hs-abarchuk @AdamOakman specFileRetries works on 6.5.0\7.0.0 versions.

[mikesalvia]

@BorisOsipov wdio-rerun-service doesn't really work in my case. I don't believe there is a way to integrate it with RP. But on the other hand the idea of having a dedicated service for RP reruns sounds interesting. Might consider building it. Thanks for bringing up this idea!

I am integrating it right now I am having a bit of trouble getting it to overwrite old results but looking into it.