Discovery on containerized edxapp rollout

[robrap]

• Some rollout questions are discussed in Discovery and cataloging of existing edxapp pipeline #920
  • Need to handle moving migrations run from the EC2 to the containerized environment (as an example).
• In a given environment (e.g. stage), what are our options for rolling out piecemeal, but also minimizing synchronization issues?
  • Will we rollout the cms webapp before lms?
  • Can we slowly add the containerized workers (cms, lms) to the mix of workers and take on small amounts of tasks before taking on more and more of the load? Are there tasks that are easy to retry and less risky to move first? Are there comms between lms and cms (shared queues) that would be better to test early?
  • What are the validation steps for each part of the migration?
    • Note: This could happen as part of this ticket, or parts of the test plan work could be ticketed separately as part of this ticket.
    • Note: We had a lack of good validation for static assets during the paver DEPR.
    • How do we validate that any performance metrics are within acceptable differences?
• For the webapp, might we want to do a route-based cutover? What would this look like? Would it require SRE? Might it just be a config change in an MFE that starts calling a different domain? Other?
• Once the migration is complete in Stage, and we learn/adjust the rollout runbook, will we want to revert and run through the rollout again in Stage to harden the runbook?
• During the overlap period, how will deployments be managed? Pauses, rollbacks, ordering of app vs. config changes (may have significantly different merge-to-prod latency)
• Note: discovery for the config-sync issue will be ticketed separately.