Rewriting the queue

[CorentinB]

The goal of this work is to replace the current queuing mechanism that use github.com/beeker1121/goque with a custom piece of code that exactly fit our needs and on which we will have much better control.

Almost all queuing components have been rewritten except the host crawl rate limiter, I commented out the relevant parts with // TODO: re-implement host limitation, @NGTmeaty if you can look into that part as you were the original author, to make it fit in the current design, it would be nice. Having it as a separate package would also be great.

I also separated the queue and seencheck packages, instead of having 1 big "frontier" package. FWIW, I'm also moving away calling it "frontier", it has only caused misunderstanding from the people I explained it too. (it was originally named like that to follow Heritrix3 path.. but let's clean things up and not use that term anymore)

Finally, I added a big list of statistics and infos about the queue that are available through the API at /queue.

Suggestions & code-review are welcome and well-needed. :)

[equals215]

lgtm, comments are mostly suggestions
good job Coco it looks really nice 😊

(i'm already crying blood from thinking I'll need to merge this to my stale PR)

[equals215]

you really sure you wanna go down the recursive path? what happens if a function calls Dequeue() when the queue is empty and mistakenly stackoverflow?

[equals215]

aligned queue structures to save space

Zeno/internal/pkg/queue/metadata.go:17:15: 16 bytes saved: struct with 224 pointer bytes could be 208
Zeno/internal/pkg/queue/metadata.go:63:14: 16 bytes saved: struct with 224 pointer bytes could be 208
Zeno/internal/pkg/queue/queue.go:30:29: 80 bytes saved: struct with 360 pointer bytes could be 280
Zeno/internal/pkg/queue/queue.go:52:11: 32 bytes saved: struct with 104 pointer bytes could be 72
Zeno/internal/pkg/queue/stats.go:8:17: 64 bytes saved: struct with 184 pointer bytes could be 120

[yzqzss]

@equals215

time=xxxxxxxxx level=WARN msg="[WORKERS] Timeout reached. 2 workers still running"

7029d7e
6a34cac

Workers get stuck in the second inner for.
revert to if statement?

[CorentinB]

@yzqzss I think --batch-write-WAL should be --batch-write-wal, but any occurence of WAL in the code in variables (such as walPath for example) should be capitalized. (as per Go variable naming convention)

[yzqzss]

@yzqzss I think --batch-write-WAL should be --batch-write-wal, but any occurence of WAL in the code in variables (such as walPath for example) should be capitalized. (as per Go variable naming convention)

yeap, I think WalIoPercent is a bit stupid too. feel free to change them yourself. :)

[equals215]

finally!

[yzqzss]

The logger has not been initialized 😱

[yzqzss]

#108