Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Verify Redis implementation #77

Open
2 tasks
anjackson opened this issue Sep 30, 2021 · 0 comments
Open
2 tasks

Verify Redis implementation #77

anjackson opened this issue Sep 30, 2021 · 0 comments

Comments

@anjackson
Copy link
Contributor

anjackson commented Sep 30, 2021

Having run a couple of heavy sets of seeds through the Redis Frontier, and after PAUSING the crawl, some queues are in qs:active and not in qs:scheduled - specifically:

0 | 1632996435109 | com,twitter,www,+1
1 | 1632996380612 | com,twitter,

and the timestamps are significantly older than those in qs:scheduled indicating that they've been forgotten. They are in qs:known so scanning that would bring it back.

  • Track down if this is a problem!
  • Determine how to properly evaluate the reliabilty of a whole replacement frontier!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant