Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Propagate launchTimestamp:XXXX annotation too #53

Open
anjackson opened this issue Feb 6, 2020 · 0 comments
Open

Propagate launchTimestamp:XXXX annotation too #53

anjackson opened this issue Feb 6, 2020 · 0 comments

Comments

@anjackson
Copy link
Contributor

When doing this:

void setLaunchTimestampInheritance(CrawlURI curi) {
// Look for refreshDepth setting:
int refreshDepth = 0; // Seeds only by default
if (curi.getData().containsKey(REFRESH_DEPTH)) {
refreshDepth = (int) curi.getData().get(REFRESH_DEPTH);
}
LOGGER.finer("Found refreshDepth " + refreshDepth + " for " + curi
+ " hop count " + curi.getHopCount());
// Check whether to inherit the launchTimestamp:
if (refreshDepth == -1 || refreshDepth > curi.getHopCount()) {
LOGGER.info("Ensuring launchTimestamp is inherited for " + curi);
curi.makeHeritable(LAUNCH_TIMESTAMP);
} else {
LOGGER.finer(
"Ensuring launchTimestamp is NOT inherited for " + curi);
curi.makeNonHeritable(LAUNCH_TIMESTAMP);
}
}

We could also add the launchTimestamp:XXXX annotation so we can see what's happened in the crawl log.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant