Add checkpoint on shutdown and --checkpoint CLI option #626
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This enables a checkpoint to be automatically created during a graceful termination. This makes it easier to stop and restart Heritrix without having to manually checkpoint each running job.
This makes it possible to select a checkpoint from the command-line when using the --run-job option.
ato
force-pushed
the
checkpoint-on-shutdown
branch
from
b4e87f4 to
ec689db
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds two features that make restarting Heritrix more convenient.
First, we add a
checkpointOnShutdownoption to CheckpointService that installs a JVM shutdown hook that creates a checkpoint before the process exits. Using the existingcheckpointIntervalMinutesoption in conjunction with it may be a good idea so that if the shutdown checkpoint was not written because the process was killed, crashed or the server lost power there is still an older periodic checkpoint to recover from.Second, we add a
--checkpointcommand-line option which gives the--run-joboption the ability to restart from a named checkpoint or the 'latest' checkpoint.When both are used together you can launch a job directly from command-line with:
Then if you Ctrl+C, Heritrix will create a checkpoint and exit. If you then run the same command again the crawl will continue from where it left off. :-)