Resource life cycle tracking

[tokee]

Some sort of life-cycle handling should be added, so that 404's or similar can be queried. De-duplication is more tricky as it seems a waste to index the same content twice. One idea is to make updatable documents by supporting some sort of pseudo-delete functionality for the frozen shards. A shared inverse filter with the IDs of deleted documents maybe? Or maybe reserve space for the delete file (and its temporary twin when deleted are added) and make a contract that says that only deletes are allowed for frozen shards?

[thomasegense]

Latest plan for revisits:
Index revisits, use hash from warc-header. It is identical to the one calculated in warc-indexer code.
new fields:
warc-id (for all records)
warc-refers-to (only for revisits)