TWDH-specific: metadata fields blanked out when running 'datapusher resubmit'

[twdbben]

Describe the bug

In the specific context of the Texas Water Data Hub schema, we run the datapusher resubmit command nightly to update resources using the following command:

ckan -c /etc/ckan/default/ckan.ini datapusher resubmit --yes

When we run this command, the 'primary_tags' and 'secondary_tags' scheming-based metadata fields on the datasets that it operates on are always blanked out when the operation completes.

To reproduce this bug, in the TWDH environment, first make sure the tag fields are set on a given dataset, then view the dataset via the API, for instance:

https://txwaterdatahub.org/api/3/action/package_show?id=surface-water-rights-water-use

Then, run the resubmit command listed above.

Wait a few seconds after the command completes - the tag-blanking does tend to lag a few seconds for some reason - and re-request the dataset from the API. Notice that the 'primary_tags' and 'secondary_tags' fields will now be empty.

[TomeCirun]

Hey @twdbben, I cannot reproduce the bug locally, I did as you suggested and everything works fine.
I worked on dev-1.0 branch + qsv 0.91.0

[jqnatividad]

Hi @TomeCirun ,
Thanks for trying to reproduce the bug on the dev-v1.0 branch usingqsv 0.91.0.

Since it will be a while till @twdbben uses dev-v1.0, can you try to repro in on DP+ 0.15.0?

It will be useful too as I'm aiming to feature-freeze 0.15.x, and that will be the basis for synching the functionality with DP+ v1.x.

[twdbben]

Thanks for investigating this @TomeCirun

Not sure if it means anything, but I find it interesting that the primary_tags and secondary_tags in your screenshot appear to be a quoted blob whereas when I look at them on my instance they are formatted as a proper list.

https://dev.txwaterdatahub.org/api/3/action/package_show?id=data-dan-s-data

Not suggesting this is the root if the problem, but perhaps this indicates something is different between our installs?

LMK if there is anything from my end that I can provide to help in replicating this issue. We are currently seeing the issue on all of our installs.

[TomeCirun]

I tested it against DP+ 0.15 and got the same result as with the dev-1.0 branch.

Regarding the type of tags, you probably have some converter for it in some of the extensions you are using. It would be useful to me if I know all the extensions(with the proper version) you are using so I can try again to reproduce the bug.

I think that the bug is not related to DP+, as the logic behind DP+ is not changing the package's metadata.

@twdbben @jqnatividad