Duplicate Datasets

[btylerburton]

Feature/what we're after

As a datagov team member, I want to understand why packages are getting duplicated during the harvest process

Anticipated/hypothesized benefits

• Our count of datasets would be accurate
• We can rely on the harvest process being atomic and idempotent

Measurements/metrics

• CKAN package count will only change when corresponding harvest sources add / remove datasets

References/background

List of fixes that have been implemented to mitigate this issue:

• restart fetch only when it is idling catalog.data.gov#548
• ...