Outcome/Objective

An MVP version of deduplication detection is implemented in the UP for ELR use cases (ORU-R01).

Context

Deduplication was originally built into the CP in response to a specific sender bug. This occurred the beginning of July 22 where a sender(PMG) introduced a bug in their system resulting in tests results not being marked as sent. That meant their system starting sending the same message over and over again. This continued for a few days before we turned off the auth for that sender. In those few days PMG was sender >800,000 messages/day. We worked with PMG and they eventually fixed the bug.

In followup to that that incident, we decided to implement deduplication logic to catch the resends in the event a similar situation occurred in the future. The idea being that duplicate reports would be discarded without the need to completely disable the sender.

Due to a bug, in the implementation the duplication feature never worked for the UP and needs to be re-implemented

Furthermore, recently to solve the file limit issue we've decoupled the batching step within the pipeline so that we can process messages one at a time. However, we do allow senders to send batched messages. Thus we will have scenarios where a sender sends a batch, some messages in that batch are good and processed and others error out. When we ask the sender to fix the errors and resend, if they send in batch, they could resend the whole batch including the messages that were already good and processed. To avoid processing the same messages twice we need a way to detect the duplicate message in the batch and ensure the duplicates are not processed, only the messages that needed fixing should get through.

Some receivers, such as CA Department of Public Health have deduplication systems in place, but not all receivers are fortunate to have the same resources. Therefore, ReportStream aims to provide the same opportunity to access to improved data accuracy by implementing deduplication prior to delivering to its receivers.

Use Cases

• As a receiver of the UP, I would like to not receive duplicative data – so that I can have a more accurate account of data that I’m receiving.

• As a sender to the UP, I would like to submit reports to ReportStream in the most convenient method possible, which may sometimes contain previously submitted, duplicate messages.

Scenarios

“Items” refer to the messages contained within a report. An item may contain multiple DiagnosticReports and Observations (including AOEs).
SCENARIOS SUPPORTED BY THIS DEDUPLICATION MVP

1. Deduplicating reports
This is what was historically supported by the COVID Pipeline.
Sender submits Report1 on Nov 1 and submits Report1 (with no changes to the content) on Nov 2. The deduplication feature will remove any subsequent submissions of Report1 (in this case, the one submitted on Nov 2).

2. Deduplicating items within the same report
This is what was historically supported by the COVID Pipeline.
Sender submits Report1 with items A, B, C, A within it. The deduplication feature will remove any subsequent instances of item A.

3. Deduplicating items across reports
Sender submits two reports: Report1 with items A, B, C & Report2 with items D, E, A. The deduplication feature will detect that item A is a duplicate across multiple submitted reports and remove any subsequent instances of item A (in this case, the one that was submitted within Report2). Items D and E will continue to be processed.

XXX SCENARIOS NOT SUPPORTED XXX

4. Deduplicating items across senders
Sender ABC submits Report1 [with items A, B, C]. Sender XYZ submits Report1 [with items A, B, C]. Both reports from both senders are exactly identical (duplicates). A deduplication feature will detect, not only all the items as duplicates, but also the reports as duplicates. Subsequent instances are removed.

5. Deduplicating observations across items
Sender submits Report1 with items A and B. Item A contains test result observations for COVID, flu, and RSV. Item B contains the same COVID and flu test result observations (for the same patient, test performed, time, etc.) and an additional observation for syphilis. A deduplication feature will detect that the COVID and Flu observations from Item A and B are identical and remove any subsequent instances (in this case, the ones from item B). The syphilis observation from Item B will continue to be processed by the pipeline.

6. Deduplicating across systems
ReportStream sends Report1 with items A, B, C to a Public Health Dpt. AIMS sends Report2 with items D, E, A. A deduplication feature (mostly likely on the PHD’s side) will detect item A as a duplicate and remove subsequent instances of it.

Product Requirements

1. Once a duplicate message is detected, prevent it from being further processed.

2. All incoming items of a report should be compared to all items submitted in at least the past year. This excludes data submitted prior to the implementation of the deduplication feature.

3. Duplication detection can be configured 'on' or 'off' for a specified sender (at the specific sender level, not sender org level). The default for senders is to have their deduplication configuration as 'on', which means deduplication is active. However, if the sender's settings for dedupe is configured as 'off', the message contents will still be hashed in event that the sender configuration will be toggled back to 'on' in the future.

4. Senders shall not be able to change their deduplication configuration without the intervention of/ request through a ReportStream admin.

5. If an Item is determined to be a duplicate, ReportStream shall log (application log AND action log) an error that an item was rejected. The log message should be: ERR, UNKNOWN, "Duplicate message was detected and removed."

6. For the MVP iteration, the deduplication feature will apply only to test results (ORU_R01) data types. However, one must keep in mind that there may be future iterations that implement deduplication for other data types, such as test orders (for ETOR).

Acceptance criteria

• All the requirements above are met in staging and production
• Run a series of tests that represent the supported scenarios above
• Each test scenario will simulate at least 5 duplicate items/ reports being detected and removed
• Confirm that the duplications are being logged correctly (syntax and timing)
• Set deduplication feature ‘off’ for all senders under the ELIMS org.
• Prior to deployment, notify Deliver and Product Mgrs of potential deduplication dependencies with active pilot sender partners.
• Technical design and final implementation are documented

Technical Design Considerations

• De-duplication in the covid pipeline currently occurs in the Receive function, but that likely does not make sense for the UP. It's worth considering if the logic in the UP should live somewhere else (i.e. convert step)

• The current logic for de-duping needs to be reconsidered for the UP; likely the item hash should be generated from the FHIR bundle so that the logic only needs to get written once - the issue here is if mappings change then FHIR bundle will be different even if original message is the same...

• What if a message fails because of a bug on our side? If they resent, would hashing see it as the same message and filter out.

• If a value is blank or not provided, consider how the hash will represent those blank fields (so that it can compare other messages with the same blank fields).

• Consider how to purge stored hashes once they have "expired" after a year.

• Should MessageID uniqueness be enforced here as well?

• In the case of SimpleReport, some unique IDs (accession number [specimen ID], CLIA number, etc.) are not guaranteed to be unique or validated due to some users of SimpleReport not having mature systems to generate and track them. If that is the case for SR, it could be true for other senders as well, making deduplication strategies less-effective or error prone. To mitigate accidentally detecting and removing non-duplicates, we recommend being more strict (comparing across many more data elements) with the deduplication criteria. This theoretically allows for more duplicates to pass through RS, but lessens the risk of accidentally removing data that is valid.

• Verify that the deduplication criteria works for all types of ELR messages (e.g., ELIMS, RADxMARS, full-ELR) and doesn't accidentally remove non-duplicates.

Criteria for an item (ORU-0R1 message) to be considered a duplicate
Combine all the following into a hash. Should the hashes match, remove the corresponding item(s) that correlate to the subsequent instance(s).