This was initially written by chimmykk and bengtlofgren. It has been optimized, bug-fixed, and enhanced with additional logic to meet the requirements of the donor drop campaign and the official frontend. This project is licensed under the MIT-license (see LICENSE).
- Contains SQL database which can be setup via Docker
- Contains scraper which is dependent on the existence of this SQL database.
npm install
OR
yarn install
Create an .env file in the root. See .env.example for an example.
Before continuing, read the comments in the docker-compose.yml file and configure this properly.
docker-compose up -d
This will setup the correct postgres database running on POSTGRES_PORT
(default port: 5434). The tables and views that will get created are specified in the init-scripts/init.sql file.
For easy access to the database, you could use a tool like pgAdmin
or dbeaver
using the credentials specified in your .env file.
Note
For a quick reset you could use the script clean-start.sh
.
IMPORTANT: this will wipe the ENTIRE database.
node scraper.mjs
Important
For now you could use a separate systemctl service to run the scraper. See issue #22 for a template.
The testing suite works as follows:
(If you have not already done so, please run npm install
and set up docker-compose as described above)
The tests will be done against the database specified in the .env
file. Ideally this would be done against a .env.test
file, but for the purposes of this project, the .env
file will be used.
To run the tests, use the following command:
npm test -- --detectOpenHandles --verbose
All tests should pass.