Keyword crawler

This script helps find content on websites. Given keywords, it scans the body tag of the html. It reports back in a csv file.

Setup

Clone repo locally
cd into main directory
install recent version of ruby if necessary
run bundle

Usage

Okay! Here is how to use this.

Make sure your list of websites are in websites.txt file, one per line
Change KEYWORDS constant in lib/website_parser.rb to point to an array of desired keywords
Script runs with ruby lib/run.rb
When finished, the results will be saved in results.csv
Note, it will also scan the url address for keywords

License

MIT.