Ingest, Temporary Files and Cron

[bseeger]

I've been testing this module and it's been great so far.

On thing I've noticed is that files get created and marked as Temporary (before I run the media ingest). Temporary files, from what I've been reading, will get cleaned up when cron runs. If a migration and cron collide (not sure how possible this is), then I can imagine that there could be an issue.

Or if I run part of an ingest and stop before I run the media portion, there's a good chance that my temporary files will be gone when I come back (if hours later) and the subsequent ingest will fail for "mysterious" reasons.

Thoughts?

[seth-shaw-unlv]

I've been bitten by this on very large loads (~1.2 TB of files) because Drupal marks temporary files to be deleted when they are 6 hours old. (Drupal deleted about a quarter of my migration before I started migrating the Media records. 🤦‍♂️) Fortunately, this is configurable. Under Home → Administration → Configuration → Media there is a "Delete temporary files after" drop-down box.

Options range from "6 hours" to "3 months" and "Never".

Bumping it up to "12 hours" was enough for me, but one could consider simply changing it to "Never" while doing migrations and changing it back once they are done.

[bseeger]

Thanks for this information. I'll play with these settings. I generally don't think this will be an issue for us, but I do think this should be explained in the documents somewhere so people are aware that it could be an issue.

[seth-shaw-unlv]

This mostly applies to migrations rather than one-off or REST-based loads (as they create files and media at the same time). It doesn't look like the Documentation site has a top-level migration page (the best place for this note, I think), just the CSV and Islandora 7 sub-pages.