TF Compare: Change script to only pull in the last 2 workflow runs (including the one that is in progress) and continue writing up bug reports when all artifacts can't be downloaded

[sihammill]

Description

Issue

Currently the TF script was changed to pull in the last 10 runs to ensure that a run with the status 'success' was picked up, however this is not always the case. If a workflow run has been successful, generated an artifact file but then failed on the TF compare script for whatever reason then we still want to use this run as in essence it was successful.

Proposed change

Only pull in the last 2 runs, we know if the run gets to the point where the TF compare script is running then it's safe to say it was a successful run.
The next change would be where the script downloads the artifact files, currently if one of the artifact files doesn't download, the script exits with an error because it can't then go an compare results. However, what we can do is still have the script search GitHub for existing bug reports for the fails from the artifact it managed to download from the current run.

Get workflow runs current code

Current code for pulling down the last 10 runs and filtering runs with a status if 'success' and 'in_progress'

response = requests.get(f"https://api.github.com/repos/{repos[1]}/actions/workflows/{workflow}/runs?per_page=10", headers=headers, stream=True)

    if response.status_code == 200:
        # Parse the JSON response
        artifact_data = response.json()

        # Filter the runs for 'success' or 'in_progress' status (conclusion)
        valid_runs = [run for run in artifact_data['workflow_runs'] if run['conclusion'] in ['success', 'in_progress']]

        # check that there is at least one run that is a success
        if [run for run in valid_runs['workflow_runs'] if run['conclusion'] in ['success']]:

            # Sort by the 'created' field in descending order to get the latest first
            sorted_runs = sorted(valid_runs, key=lambda x: datetime.fromisoformat(x['created_at'].replace('Z', '+00:00')), reverse=True)

            workflow_runs = []

            # Add the latest 2 runs to an array
            for run in sorted_runs[:2]:
                workflow_runs.append(run)

Refactored code

This will only pull down the previous 2 runs regardless of status

response = requests.get(f"https://api.github.com/repos/{repos[1]}/actions/workflows/{workflow}/runs?per_page=2", headers=headers, stream=True)

    if response.status_code == 200:
        # Parse the JSON response
        artifact_data = response.json()

        workflow_runs = artifact_data.get("workflow_runs", [])

Current code for getting each .json file from the artifact zip

This is fine if the artifact exists, if it doesn't, the script will crash out

# for each json file
        for art_file in artifact_files:
            
            # json file location
            file_path = f"{baseSaveLocation}/{data}/{art_file}"
            # open and read the json file
            with open(file_path, 'r') as file:

                #if fileCount == 0:
                # eg. xUnit_windows_VM_1, xUnit_windows_YYC_1 - Latest Test Run
                #     xUnit_windows_VM_2, xUnit_windows_YYC_2 - Previous Test Run
                allTestFiles[f"{art_file}_{fileCount}"] = json.load(file)

        fileCount +=1

Refactored code

Adding in a new line to check if the file actually exists eliminates the script from crashing and allows the script to continue

# for each json file
        for art_file in artifact_files:
            
            # json file location
            file_path = Path(f"{baseSaveLocation}/{data}/{art_file}")
            # check the file exists
            if file_path.exists():
                # open and read the json file
                with open(file_path, 'r') as file:

                    #if fileCount == 0:
                    # eg. xUnit_windows_VM_1, xUnit_windows_YYC_1 - Latest Test Run
                    #     xUnit_windows_VM_2, xUnit_windows_YYC_2 - Previous Test Run
                    allTestFiles[f"{art_file}_{fileCount}"] = json.load(file)
            else:
                print(f"File in {file_path} does not exist.")
        fileCount +=1

Conclusion

The script should never fall over, the only time it should exit and not complete is if an artifact in the current run hasn't been created, however, if that is the case, the TF compare script will never start to run in the first place because the workflow would have already failed before reaching the TF compare script.

If in the last 2 runs (current run included) have 2 valid artifact files, they will be downloaded and compared, if the previous run doesn't have an artifact file because maybe the run crashed then the script will continue to download the artifact file from the current run and report the fails as bug reports as usual.

Script Name

tf_compare.py

Steps To The Task

• Have the script only pull down the last 2 runs, 1 will be be the current run
• Ensure the script continues with writing up new bug reports for the current fails, even if the artifact for the previous run can't be downloaded.