Implement design to handle queued events on policy update

[leehinman]

Problem Statement

When a new policy is sent to the shipper, it is possible that the new policy will remove configuration that would apply to events that are in the queue (especially disk queue).

Design

When a shipper starts it should save a list of destination data streams it has configuration for. Outputs should check events against the list and log (DEBUG level) and drop any events that it doesn't have configuration for.

When a new policy is received by the shipper, the following should happen:

1. stop receiving new events from inputs
2. outputs should stop reading from the queue
3. wait with deadline for responses from in-flight writes to output to
   give them time to complete.
4. send any ACKs to inputs
5. reload with new configuration
   • as part of reload update the list of destination data streams
   • as part of reload start sending via outputs (which will check against updated list)
   • as part of reload start reading new events via inputs

Other Docs

• Historical Design Document
• Correctly handle enqueued events affected by agent policy changes #49