CKAN duplicate dataset (same harvest object)

[jbrown-xentity]

Perfect duplicate (other than name):

• https://catalog.data.gov/api/action/package_search?q=identifier:%2242c22469-3c8b-415a-86d2-e38bbd192e5d%22
• https://catalog.data.gov/dataset/observed-precipitation-temperature-and-snow-water-equivalent-for-the-rio-grande-headw-1905-267a5
• https://catalog.data.gov/dataset/observed-precipitation-temperature-and-snow-water-equivalent-for-the-rio-grande-headw-1905-e2738

These even have the same harvest object id, but are harvested almost 4 hours apart from each other (but on the same job, at the time of this analysis only 1 harvest job has been run). There is no reason for this to occur, the code should be preventing it. The only reason I can imagine is if the harvest fails to mark this object as "complete", but it runs through everything up to that point (including actually creating the dataset), and it somehow re-enters the queue.

How to reproduce

1. Unknown

Expected behavior

No duplicate datasets are harvested from a DCAT source (by design)

Actual behavior

Duplicates are created

Sketch

The goal of this ticket is to investigate and propose a resolution to this problem. This is a different outcome (the datasets are created in CKAN), but may be related to #3943.

This may involve digging into logs during this time period for this dataset, trying to resolve how this record was parsed twice from the queue.

Once the solution is in place, a follow up will need to be done/ticket be created to

1. Discover how many items this affects
2. Clean up any affected records

This may require creating custom processes/scripts in https://github.com/GSA/datagov-dedupe

[jbrown-xentity]

After examining logs using catalog-fetch observed-precipitation-temperature-and-snow-water-equivalent-for-the-rio-grande-headw-1905 for time period from Sep 1 6am - Sep 2 6am, we found that 3 objects were created. The first and final objects print 5 times each, with a notes corresponding with:

• Starting package_create
• Indexing
• Package_create complete
• Starting re-index
• Re-index complete

The 2nd object only logged the Starting package_create log, nothing else. Need to investigate further on what may have happened during/around the 2nd statement at 2022-09-01T16:26:06.157Z on the 0 instance to see what errors or commands may have halted the import.

[btylerburton]

Determined that a restart of the container occurred directly after 2022-09-01T16:26:06.157Z. With the recent fix to only restart fetch process when idle, we expect this issue to resolve itself.

GSA/catalog.data.gov#548

[jbrown-xentity]

We have seen this occur again. See list of DOI duplicates here: https://catalog.data.gov/api/action/package_search?fq=organization:doi-gov&facet.field=[%22identifier%22]&facet.limit=-1&facet.mincount=2&rows=0
We have confirmed that these datasets exist, with the same harvest object, are duplicated in the DB and SOLR, after the fix above has been in place.
Re-opening...

[jbrown-xentity]

We believe that with #4007 being complete, as well as the previous work on this ticket, that this may be resolved. Should examine DOI as an example, and the list of all duplicates from here against previous runs to check if any counts are growing. If the counts are static/down, this can be closed. If growing, we should investigate further.

[jbrown-xentity]

I have spot checked various current duplicates, and find a different case than this one. Closing.

[FuhuXia]

Found a newly created one, same harvest object id, 14 seconds apart on creation time.

https://catalog.data.gov/api/action/package_search?fq=(identifier:%22https://doi.org/10.23719/1526064%22)

dataset 1
dataset 2

[jbrown-xentity]

The relevant logs should be here. I don't see anything that jumps out, whether instances restarting or suspicious failures. It's possible that it was inserted into the queue twice from the gather? Or maybe something else weird is going on... gather logs.

[FuhuXia]

We can manually create a duplicate by inserting the same harvest_object_id into Redis DB twice. For datajson source, it will create a duplicate dataset, for XML sources, it will generate an error Validation Error: {'Name': 'That URL is already in use.'} that we saw in the other tickets, but it won't create a duplicate. So the issue is most likely due to glitches between multiple fetch processes and the Redis.

[FuhuXia]

Commit GSA/ckanext-harvest@f2f97f4 should prevent duplicate dataset (same harvest object) from happening. Will move it further if this issue is happening more frequently. So far it only happened once, not giving us enough logs to analyze to find a general pattern. We can use this API call to do regular check.

https://catalog.data.gov/api/action/package_search?facet.field=[%22harvest_object_id%22]&facet.limit=-1&facet.mincount=2&rows=0&fq=(collection_package_id:*%20OR%20*:*)

Will manually run the dedupe process to remove the found one, then close this ticket.

[FuhuXia]

Moving back to in progress since we found 83 cases during the past two weeks.

[FuhuXia]

The 83 has resolved on its own, mostly likely due to the recent deployed db-solr-sync with harvest_object_id.

[jbrown-xentity]

I found another one, not sure if it will resolve on it's own or not (but I believe it persisted from yesterday): https://catalog.data.gov/api/action/package_search?q=identifier:%22https://doi.org/10.23719/1526064%22&facet.field=[%22harvest_object_id%22,%20%22harvest_source_title%22]