Streamlining the exception process to submit to stabilization during a release period

[AMZN-daimini]

During the retrospective on the 22.10 release, multiple parties have raised the concern that our current exception process to push changes to the stabilization branch during a release period can be confusing or cumbersome to execute.

The current process is outlined here: #43

Here is a summary of the issues that were raised after the past releases:

1. Discovery - Users that don't follow the updates and announcements on the Discord have a hard time understanding that they need to seek an exception, or how the process works. This has resulted in changes being merged without going through the process as intended, or changes going in after the release deadline (when the stabilization branch was no longer going to be merged back to main, resulting in those changes to get lost).
2. Tracking - When the exception process has started, a pretty advanced level of knowledge of the organizations is necessary to navigate the steps of the checklist. For example, there is no way to know what sig the people that gave approvals to the PR are representing, and if they are the chair or a contributor that has the power to give approval for their sig, so most contributors need the support of a sig-release member or a maintainer to determine whether they have achieved the requirements or not.
3. Notification - It is hard for sig chairs to track PRs in need of their approval for the exception process. This usually led to sig chairs being mentioned directly either on the PRs themselves or Discord, causing noise and making it harder to organize the work.

As these issues are interconnected, we determined to have a discussion about possible solutions before determining which path(s) to follow and creating precise tasks. I will add some of the possible solutions that were discussed during sig-release meetings; feel free to comment on them or add new suggestions.

[AMZN-daimini]

Regarding point 1. Discovery
It could be useful to have a bot automatically trigger whenever a user creates a PR that has the stabilization branch as the destination.
The Bot could comment on the PR as soon as it's created and detail the next steps depending on the current release period:

• In the early stabilization phase, the bot can redirect the user to submission guidelines. If the PR is for a bugfix, no further action is required; if the change is a new feature, they should have a way (either automatically via the bot, or via copy-pasting instructions) to initiate the exception process seamlessly
• In the later stabilization phase, the bot should just automatically initiate the exception process by adding the correct labels and adding a comment with the checklist.
• After the stabilization deadline, the bot could post a comment stating that the change will not be merged back to the release nor development automatically if it's pushed, and should be retargeted.

[AMZN-daimini]

Regarding point 2. Tracking
It is currently hard to keep track of which requirements for the checklist have been met and which haven't, since that requires knowledge of who the chairs of each sig are.
A possible way to streamlining the process could be to automate part of the verification of the requirements by better defining the roles within GitHub:

• Each SIG should define a group (similar to the maintainers and reviewers groups) for people that can authorize a PR during the exception process for that sig. With the current guidelines, that group would only contain that sig's chairs but having a more generic group name may help if that pool needs to be expanded down the line.
• Use the groups defined above in conjunction to a bot to automatically detect which sigs have given the PR the approval. This should remove the need for sig chairs to manually comment on the PR stating the sig they represent.
• (advanced / not MVP) the bot could either provide controls to define which sig approvals are required, or automatically detect it, and add those requirements to the list of steps needed to enable the merge button.

[AMZN-daimini]

The problem with having automated verification via groups is that it adds a level of maintenance that we don't currently take care of. Someone would need to ensure groups are automatically updated whenever sig chairs change, or if the role gets expanded, make sure people stepping out are removed from the group in a timely manner.
If the automation side is technically doable, a discussion on the operations side should also be included before moving on with this.

[AMZN-daimini]

Regarding point 3. Notification
I'm not sure how much automation can help without just adding additional noise, which would revert us back to the original issue.
One possible solution could be to have an automated process to generate a daily newsletter with a list of all PRs in the exception process and their status, so that sig chairs can subscribe to that newsletter and get those notifications condensed in one place instead of having to follow up on multiple callouts from different places. Bonus points if it can link to the live board that was already manually updated.