Core processor logic (orphan data problem)

[skeating]

The HL7 messages that Emap has been ingesting up to this point tend to be highly redundant. Every message contains basic patient demographic/encounter data. This means that if we receive messages out of order, keeping the database in a usable state at all intermediate points is not too hard. For example, if we receive a test order for a patient that we haven't yet received any other information for (admission, demographics, etc) we can still create the encounter/mrn/demographics from that message alone.

Unfortunately this is not the case for waveform data. The only patient ID info we have in a waveform HL7 message is the bed location, the time, and (presumably) the machine identifier. This raises the possibility that we will receive some waveform data and have no idea which encounter to attach it to.

NB: Although it may seem unlikely that a patient would start generating ventilator/monitor data before we have seen their admit messages, consider what happens if the main HL7 feed goes down for a bit but the waveform feed doesn't.

I can see several ways around this problem:

• Just reject the data if we don't know who is in that location
  • pro: simple
  • con: we lose data
• Write the "orphan" waveform data to the database with its bed location, but without a link to a visit/encounter, then fix it up later
  • pro: eventually correct
  • cons
    • will need careful consideration as to when to perform this check
      • every time a new visit is opened, see if there is some orphan waveform data waiting for it?
      • or just intermittently check for it?
    • Emap users will have to detect and exclude orphan data
• Don't ever try to link the waveform data to a hospital visit
  • pro: really easy for us!!
  • cons: terrible for the user, this is not the Emap way

We currently don't store any orphan data in Emap, so this will be a new paradigm for us.

A closely related problem:

• Patient A is assigned to bed 1
• The main HL7 feed goes down
• Patient B is assigned to bed 1
• We receive waveform data for bed 1, and wrongly assign it to person A

We may need to actively monitor the overall liveness of our data, and perform whatever fix we come up with for this problem in this case.
Possible heuristics to answer the question "Is patient A definitely still assigned to bed 1?":

• Have any ADT events happened (to anyone) in the database within the last few minutes? May not be a good measure in the middle of the night, when waveform data will still be coming in.
• We have recently (last 1 minute?) received waveform data for bed 1 that we assigned to patient A. Even if the feed has gone down, there's no way that machine switched patients so quickly.

Other solution is to fix it up later when the feed comes back, but that involves a lot of continuously rechecking stuff, and we will have been giving out wrong information in the meantime. And then we will need that audit table!

[jeremyestein]

Closing because this is a design discussion not a task, and to the extent it's an epic I have merged it with #27

For design discussion see https://github.com/UCLH-DHCT/emap/blob/jeremy/hf-data/docs/dev/features/waveform_hf_data.md