Notification system design (ie, Email)

[mdial89f]

I wanted a place to discuss our ideas for how notifications will work in micro.

Background

Notifications are critical to the function of our application. The notifications sent by our app are how CMS personnel will be alerted that something was submitted or assigned. They will notify the state submitter of events, too.
New submissions and all current post submission actions should send a notification/notifications, usually to someone at CMS and someone at the state.

Assumptions

We'll assume that the only end-user notification type is Email. While most of our solutions can support any type of notification, Email is what's historically been used and the only foreseeable ask.

Design Goals

This section will change as we discuss. This is a starting point.

• Fault Tolerant: temporary failures such as network should be handled.
• Retryable: a human should be able to resend a notification for an event.
• Auditable: a human should be able to quickly see what emails have been sent, with a means to search.
• Monitored: the notification system should report health metrics to our cloudwatch dashboard, and be hooked to our internal SNS system notification system

Discussion...

The design goals above and ideas on how to technically approach achieving them are the main parts of this discussion. Input regarding current system pain points and user experience is welcome, and could shift our background and/or assumptions.

Please jot down any and all thoughts or ideas. This is the space for it. Thanks

---

This space is reserved for the outcome of this discussion. I'm not sure of the form of the output, but I think a list of recommendations with regard to technical approach is a good target.

[kristin-at-theta]

Some notes about the current implementation:

• uses AWS SES as an account service with all stacks sending emails through it. I think this is necessary because of the need for a verified sending address, but someone may wish to verify that.
• embeds email sending within the action workflow because the successful sending of the CMS email is a condition for submission success (the sending of the state email is not).
• sometimes makes queries on the database in order to build the email. For example, when a state user withdraws a package, ALL state submitters are sent the receipt. (example)
• most sending failures are from submit lambda timeouts (the database dependency on a "eventual" database, DynamoDB, looks like Mako will not have that problem (the little popup saying the data isn't real time))

[Dark-Knight-1313]

Please review a proposed designed document to address failed emails retries.email retries

[mdial89f]

Hello, I'm recapping a meeting that was just had with @kristin-at-theta @Dark-Knight-1313 @benjaminpaige @mdial89f @Valencia2019 @bflynn-cms

We spent about an hour discussing various approaches to sending emails.

---

The consensus leaving the call was:

High level architecture: Email lambda subscribed to Kafka topic
Replay: Regenerating email based on event data
Addressee List: Querying cognito to get all users for a state

Details below

---

Questions and Considerations:

• Where is the data that's needed for each email contained
• Where do we store email templates
• How do we resend emails
• How often do we resend emails
• How do we ensure delivery
• How do we accommodate changing templates over time
• Who do we send emails to

High level architecture options

These options speak to how we're going to organize the appliance that sends email, how it will its template and event data, and how its triggered.

Synchronous Invocation from lambda

This solution invokes SES from the lambda backing the action's API endpoint. For instance, responding to an RAI hits an api endpoint, which triggers a lambda. In this scenario, that lambda would do the email generation based on the event data, and invoke SES to send the email before exiting.

Pros:

• Sending the email during the api's lambda lifecycle is straightforward. If something occurs calling SES, the lambda will fail, and we have an obvious opportunity to return that failure to the application.

Cons:

• Considerably expands the scope of the API action lambda

Decoupling with SQS or SNS

This would have the API action lambda doing its business logic for the event, and then sending that event data to an SQS queue or SNS topic. The event would then be processed by an lambda specifically built for email generation and sending, and sent out.

Pros:

• Lower impact on the api action lambda's scope than alternatives
• The decoupling prevents temporary network failures related to sending the email from failing the api invocation.
• Standard error handling patterns exist for this solution, such as dead letter queues.

Cons:

• Complexity. New infrastructure to build, maintain, and monitor.

Email lambda subscribed to Kafka topic

Here, a lambda specifically built to generate and send emails is created. The lambda is subscribed to the onemac kafka topic, as an event source. When the lambda is triggered by events passed to it by aws, it generates the email from the event data and template, and sends it. Retry would entail taking a past event from package activity and sending it back to the email lambda.

Pros:

• No impact to the API; completely isolated service
• No new event data to transmit or read, it's reusing what's already sent to kafka
• Reusing msk native event source mapping

Cons:

• Each event needs to contain all information to fill in the templates; need to verify this

Replay options

There is sometimes a need to resend an email for a past event. This can be due to send failure, user error (i deleted the email), or request. These options are how to support resending an email.

Storing static email for later recall

This solution would have the lambda emailer saving a copy of the final state, substituted template in a data store, perhaps S3. If an event needs an email resent, an appliance would recall that object and send it.

Pros:

• There's no reliance on past event data and the current template to be compatible.

Cons:

• Overhead and complexity; storing all emails and building an appliance to programmatically recall the right one and resend has some complexity that should be considered.

Regenerating email based on event data

This solution calls for a past event, like responding to an rai, to have it's data resent to the email lambda, where it's passed into a template and resent.

Pros:

• This is a consistent approach to how emails are sent when the event occurs, with less infrastructure to build.

Cons:

• Old event data and new templates could be incompatible if breaking changes are made.

Addressee List options

There's a specific requirement for the old system that we expect to hold for the new. On certain events, all state submitters for the package's state should be notified. The question is how to best meet that requirement.

Maintaining a list of all users for a state

Here we would probably tie a lambda to the cognito auth event. When a state submitter logs in, we look at their states, and add their email to the states for which they have access. Dynamo or another store could be leveraged to hold this state. The emailer lambda can recall this list when it needs it.

Pros:

• Fast access to a list of all state submitters, as it's kept up to date as people log in.

Cons:

• Overhead and complexity. We don't have dynamo in our application yet, and avoiding a novel data store should be considered.

Querying cognito to get all users for a state

Here the emailer lambda would query the cognito user pool to get the emails of all state submitters for a given state.

Pros:

• No additional datastore to build and maintain

Cons:

• Performance. If this query ends up being too long or complex, it may not be practical.

[kristin-at-theta]

Ran into a snag.... the submission event does not hold all the required data for the emails to be created. Specifically, the proposedEffectiveDate in the current work. We also are looking ahead and know we'll need the CPOC email, the SRT list, and possibly other original package details for the withdrawal action and RAI response emails.

For the completion of the current story... can either:

• remove the inclusion of the Proposed Effective Date in the email
• add Proposed Effective date to the action event so it gets passed to the email sender

Long term solution: Use a submission event to trigger the email sending, but query the OpenSearch data to get the details.

Because SEA Tool updating can be delayed, makes sense to use the SEA Tool event as the triggering event for the emails. Though this places a dependency on SEA Tool being up for the email to go out, which, since the emails are supposed to help catch problems with SEA Tool, might not be the best idea.

[mdial89f]

@kristin-at-theta Great info, unfortunate snag.
I'm not offering a solution, just thinking on things. But I'll point out one bit to consider:

The seatool event stream is populated by ksqldb. Ksqldb works to roll up several other streams any time a change is made. Due to its functioning, a single update in seatool may emit many events. I think we've seen new submissions generate 7 events. This behavior isn't a major problem, because ksqldb dosn't output discrete events... it outputs the materialized view of the record, or the entire record every time. So multiple events in a row aren't problematic for our data sinks. However, triggering a notification off of it is likely incompatible. Without knowing the before state of the seatool record, you can't resolve what change occurred. And if you do, you'd then have to deal with 'n' number of events. All to say: off hand, I think a trigger off the seatool stream may prove unworkable.

[kristin-at-theta]

ok ... then, is it possible to add the proposed effective date as an attribute from the submit event?... and if that causes some sort of primary race condition based on which event happens when, could we name it something else?

[kristin-at-theta]

Also wondering how Mako is determining the "90th Day Deadline." Current OneMAC does a calculation off of Initial Submission Date, but Initial Submission Date is another data element that does not come through the event. I am not a fan of doing it in the email creation, especially if we have a centralized way to calculate the deadline date. I suppose I could put it in the shared-utils?

[kristin-at-theta]

There is really good conversation about this subject in: #377