[Alerts] Create Datadog alert for edx-platform edge to learn about alerting

[timmc-edx]

A/C:

• One edxapp Edge alert policy migrated from NR to DD
  • Include both performance and error related conditions—prod-edge-edxapp-lms-arch-bom covers this.
  • May not need to transfer everything over—some conditions are duplicates or have been disabled.
• New alert is informed by DD best practices
• How-to documentation created or updated for alert migration
• Team alerts runbook updated
  • Moved this work to Migrate Arch-BOM alerts/monitors #662

We'll use Edge because it has DD enabled already and actually has traffic.

---

Created monitors

Error rate

• https://app.datadoghq.com/monitors/144647801
  • Error burst, same params as in New Relic
  • Multi-monitor across all environments, but env:stg is downgraded to a P2
• https://app.datadoghq.com/monitors/145005984
  • Same as above, but for sustained lower rate

Apdex

Datadog doesn't have to a way to break out apdex by custom span tags, so I reimplemented the apdex formula in Trace Analytics.

Given a base filter env:(prod OR edge) service:edx-edxapp-lms operation_name:django.request @code_owner_squad:arch-bom (over service entry spans) we can define three queries:

• a: <BASE> [email protected]_code:5* @duration:[0ms TO 400ms]
• b: <BASE> [email protected]_code:5* @duration:[401ms TO 1600ms]
• c: <BASE>

And then alert on (a + b/2) / c < 0.8 over 15 minutes: https://app.datadoghq.com/monitors/145167039 Here I've chosen 400 ms as the apdex T value; we're using 2 seconds in New Relic but that's way too high. I set 0.85 as the recovery threshold to add some hysteresis.

[robrap]

@timmc-edx: I started a Slack thread with DD about getting a best practice doc. Also, it looks like @alangsto started some how-to docs on monitors here.

@alangsto: [inform] Tim is working on arch-bom alerts in this ticket.

[robrap]

@timmc-edx: I'm wondering if these supporting runbooks have dashboards, and if it would be useful migrate those dashboards as part of this ticket, or if we should spin off a separate ticket for dashboards. Let me know your thoughts. Thanks.

[timmc-edx]

I think we should ticket dashboards separately.

[timmc-edx]

Notes:

These conditions are currently enabled in the NR policy; the disabled ones are "baseline" error rate alerts that we replaced with static threshold alerts:

• BOM Edge LMS APM Apdex Burst
• BOM Edge LMS APM Apdex Sustained
• BOM Edge LMS APM Error Burst (1% over 10 min)
• BOM Edge LMS APM Error Sustained (0.1% over 60 min)
• BOM Edge LMS Workers High Error Rate

Pattern: Burst/sustained are the same query with different thresholds. All filter on code owner being arch_bom or NULL.

[timmc-edx]

I was using @http.status_code:5* for the error monitors but have switched to status:error to make it more general. However, these produce nearly identical results, as status:error is only being evaluated on the service entry span. When an error bubbles up that far, we produce a 500 anyhow. (There are a handful of traces that have status:error and no response code but these are probably just incomplete traces.)

[robrap]

What is an incomplete trace, and is it the result of a bug? Can we ask DD about this, possibly in a support ticket?

[timmc-edx]

I can make a note to look into it further.