High CPU usage

[ocervell]

cariddi can give crazy CPU usage and decent memory usage when run on a huge list of URLs (~150 in this case).
We should probably profile it and find areas of improvements for performance. Such a library should be I/O heavy but not too CPU or memory heavy normally.

cat urls.txt | cariddi -info -s -err -e -ext 1 -json -c 5

[edoardottt]

Hi @ocervell ! Thanks for the issue.
I know it's quite resource intensive since the first commits :(
I guess the CPU and memory high load comes from pattern matching.
I'd love to profile it and understand if we can improve it, however I've never used a profiler.

Let me know if you can help in some way, I'd love to improve cariddi.

Thanks again