Migrate Arch-BOM alerts/monitors

[dianakhuang]

Acceptance Criteria:

• go through the spreadsheet
• migrate the alerts and alert policies to DataDog
• update runbooks to point to DataDog

Policies:

• prod-edx-edxapp-lms-arch-bom - Mostly done as part of [Alerts] Create Datadog alert for edx-platform edge to learn about alerting #628 but will be converted to Terraform; runbook needs update @timmc-edx
  • create monitors (lms)
  • create monitors (lms-workers)
  • update runbook
  • review monitors and runbook
• prod-edge-edxapp-lms-arch-bom - ditto @timmc-edx
  • create monitors (lms)
  • create monitors (lms-workers)
  • update runbook
  • review monitors and runbook
• prod-edxapp-lms-arch-bom-backend-safety-net @dianakhuang
  • create monitors (in-progress)
  • update runbook
  • review monitors and runbook
• platform-arch-bom-event-bus-safety-net @robrap
  • create monitors (logs) (in-progress)
  • ticket non-log monitors and runbooks (moved to Ensure Kafka Event Bus APM monitoring #658)
  • review monitors and runbook
• platform-arch-bom-ownership
  • create monitors
  • update runbook
  • review monitors and runbook
• platform-arch-bom-code-owner-issues - @timmc-edx
  • create monitors
  • update runbook
  • review monitors and runbook

After runbooks have been migrated:

• Archive https://2u-internal.atlassian.net/wiki/spaces/ENG/pages/19470837/Squad+Owned+LMS+New+Relic+Alert+Runbook

[robrap]

For event bus monitoring, migrate the following:

• Consumer error logs across multiple topics [P3]
• Producer error logs across multiple topics [P3]

Move the following monitors to the Kafka ticket:

• No event consumed in past day on any topic [P3]
• No event produced in past day on any topic [P3]
• TransactionErrors in consuming across topics [P3]

[timmc-edx]

Still to do:

• Migrate code-owner-issues alerts
• Perform a review on monitors and runbooks