Webpage crawler. I recommend to run sample rails app from ./sample_app and test against it

Usage: ./crawler [options]
    -u, --url STRING                 Base URL to fetch
    -c, --concurrency INTEGER        Number of concurrent crawling threads. Default: 4
    -p, --pages INTEGER              Max number of pages to fetch. Default: 50
    -d, --depth INTEGER              Max page depth relative to base url. Default: 3
    -h, --help                       Display this help