new victims monitoring is broken, alert only when sites are down

[Deventual]

Describe the bug
The app doesn't alert when new victims added to the ransom sites (we noticed that new victim are being added on some of the sites)
We get alerts only when the sites are down.

Expected behavior
The app alert when new victim are added to the ransom sits being monitored.

Logs
Starting ransomwatch_proxy_1 ... done
Starting ransomwatch_app_1 ... done
Attaching to ransomwatch_proxy_1, ransomwatch_app_1
proxy_1 | Feb 07 14:50:31.819 [notice] Tor 0.4.5.7 running on Linux with Libevent 2.1.12-stable, OpenSSL 1.1.1i, Zlib 1.2.11, Liblzma 5.2.5, Libzstd 1.4.5 and Unknown N/A as libc.
proxy_1 | Feb 07 14:50:31.822 [notice] Tor can't help you if you use it wrong! Learn how to be safe at https://www.torproject.org/download/download#warning
proxy_1 | Feb 07 14:50:31.822 [notice] Read configuration file "/etc/tor/torrc".
proxy_1 | Feb 07 14:50:31.825 [notice] Opening Socks listener on 0.0.0.0:9050
proxy_1 | Feb 07 14:50:31.825 [notice] Opened Socks listener connection (ready) on 0.0.0.0:9050
proxy_1 | Feb 07 14:50:31.825 [notice] Opening Control listener on 0.0.0.0:9051
proxy_1 | Feb 07 14:50:31.825 [notice] Opened Control listener connection (ready) on 0.0.0.0:9051
app_1 | 2022/02/07 14:50:33 [INFO] Initializing
app_1 | 2022/02/07 14:50:33 [INFO] Found 30 sites
app_1 | 2022/02/07 14:50:33 [INFO] Starting process for Avaddon
app_1 | 2022/02/07 14:50:33 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:50:33 [INFO] Starting process for Conti
app_1 | 2022/02/07 14:50:38 [INFO] Scraping victims
app_1 | 2022/02/07 14:51:48 [INFO] There are 0 new victims
app_1 | 2022/02/07 14:51:48 [INFO] Identifying removed victims
app_1 | 2022/02/07 14:51:48 [INFO] There are 0 removed victims
app_1 | 2022/02/07 14:51:48 [INFO] Finished Conti
app_1 | 2022/02/07 14:51:48 [INFO] Starting process for DarkSide
app_1 | 2022/02/07 14:51:48 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:51:48 [INFO] Starting process for REvil
app_1 | 2022/02/07 14:51:48 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:51:48 [INFO] Starting process for Babuk
app_1 | 2022/02/07 14:51:50 [INFO] Scraping victims
app_1 | 2022/02/07 14:51:51 [INFO] There are 0 new victims
app_1 | 2022/02/07 14:51:51 [INFO] Identifying removed victims
app_1 | 2022/02/07 14:51:51 [INFO] There are 0 removed victims
app_1 | 2022/02/07 14:51:51 [INFO] Finished Babuk
app_1 | 2022/02/07 14:51:51 [INFO] Starting process for Ranzy
app_1 | 2022/02/07 14:51:51 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:51:51 [INFO] Starting process for Astro
app_1 | 2022/02/07 14:51:51 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:51:51 [INFO] Starting process for Pay2Key
app_1 | 2022/02/07 14:51:53 [INFO] Scraping victims
app_1 | 2022/02/07 14:51:54 [INFO] There are 0 new victims
app_1 | 2022/02/07 14:51:54 [INFO] Identifying removed victims
app_1 | 2022/02/07 14:51:54 [INFO] There are 0 removed victims
app_1 | 2022/02/07 14:51:54 [INFO] Finished Pay2Key
app_1 | 2022/02/07 14:51:54 [INFO] Starting process for Cuba
app_1 | 2022/02/07 14:51:57 [INFO] This is the first scrape for Cuba, no victim notifications will be sent
app_1 | 2022/02/07 14:51:57 [INFO] Scraping victims
app_1 | 2022/02/07 14:52:08 [INFO] There are 0 new victims
app_1 | 2022/02/07 14:52:08 [INFO] Identifying removed victims
app_1 | 2022/02/07 14:52:08 [INFO] There are 0 removed victims
app_1 | 2022/02/07 14:52:08 [INFO] Finished Cuba
app_1 | 2022/02/07 14:52:08 [INFO] Starting process for RansomEXX
app_1 | 2022/02/07 14:52:10 [INFO] Scraping victims
app_1 | 2022/02/07 14:52:13 [INFO] There are 0 new victims
app_1 | 2022/02/07 14:52:13 [INFO] Identifying removed victims
app_1 | 2022/02/07 14:52:13 [INFO] There are 0 removed victims
app_1 | 2022/02/07 14:52:13 [INFO] Finished RansomEXX
app_1 | 2022/02/07 14:52:13 [INFO] Starting process for Mount
app_1 | 2022/02/07 14:52:13 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:52:13 [INFO] Starting process for Ragnarok
app_1 | 2022/02/07 14:52:13 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:52:13 [INFO] Starting process for Ragnar
app_1 | 2022/02/07 14:52:13 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:52:13 [INFO] Starting process for Suncrypt
app_1 | 2022/02/07 14:52:15 [INFO] This is the first scrape for Suncrypt, no victim notifications will be sent
app_1 | 2022/02/07 14:52:15 [INFO] Scraping victims
app_1 | 2022/02/07 14:52:17 [INFO] There are 0 new victims
app_1 | 2022/02/07 14:52:17 [INFO] Identifying removed victims
app_1 | 2022/02/07 14:52:17 [INFO] There are 0 removed victims
app_1 | 2022/02/07 14:52:17 [INFO] Finished Suncrypt
app_1 | 2022/02/07 14:52:17 [INFO] Starting process for Everest
app_1 | 2022/02/07 14:52:17 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:52:17 [INFO] Starting process for Nefilim
app_1 | 2022/02/07 14:52:17 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:52:17 [INFO] Starting process for Cl0p
app_1 | 2022/02/07 14:52:17 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:52:17 [INFO] Starting process for Pysa
app_1 | 2022/02/07 14:52:19 [INFO] Scraping victims
app_1 | 2022/02/07 14:52:23 [WARNING] couldn't parse timestamp: 00/00/00
app_1 | /usr/local/lib/python3.9/site-packages/dateparser/date_parser.py:35: PytzUsageWarning: The localize method is no longer necessary, as this time zone supports the fold attribute (PEP 495). For more details on migrating to a PEP 495-compliant implementation, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html
app_1 | date_obj = stz.localize(date_obj)
app_1 | 2022/02/07 14:52:24 [INFO] There are 0 new victims
app_1 | 2022/02/07 14:52:24 [INFO] Identifying removed victims
app_1 | 2022/02/07 14:52:24 [INFO] There are 0 removed victims
app_1 | 2022/02/07 14:52:24 [INFO] Finished Pysa
app_1 | 2022/02/07 14:52:24 [INFO] Starting process for Hive
app_1 | 2022/02/07 14:52:24 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:52:24 [INFO] Starting process for Lockbit
app_1 | 2022/02/07 14:52:24 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:52:24 [INFO] Starting process for Xing
app_1 | 2022/02/07 14:52:24 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:52:24 [INFO] Starting process for Lorenz
app_1 | 2022/02/07 14:52:26 [INFO] This is the first scrape for Lorenz, no victim notifications will be sent
app_1 | 2022/02/07 14:52:26 [INFO] Scraping victims
app_1 | 2022/02/07 14:52:27 [INFO] There are 0 new victims
app_1 | 2022/02/07 14:52:27 [INFO] Identifying removed victims
app_1 | 2022/02/07 14:52:27 [INFO] There are 0 removed victims
app_1 | 2022/02/07 14:52:27 [INFO] Finished Lorenz
app_1 | 2022/02/07 14:52:27 [INFO] Starting process for ElCometa
app_1 | 2022/02/07 14:52:27 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:52:27 [INFO] Starting process for Arvin
app_1 | 2022/02/07 14:52:30 [INFO] This is the first scrape for Arvin, no victim notifications will be sent
app_1 | 2022/02/07 14:52:30 [INFO] Scraping victims
app_1 | 2022/02/07 14:52:33 [INFO] There are 0 new victims
app_1 | 2022/02/07 14:52:33 [INFO] Identifying removed victims
app_1 | 2022/02/07 14:52:33 [INFO] There are 0 removed victims
app_1 | 2022/02/07 14:52:33 [INFO] Finished Arvin
app_1 | 2022/02/07 14:52:33 [INFO] Starting process for Blackmatter
app_1 | 2022/02/07 14:52:33 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:52:33 [INFO] Starting process for Avoslocker
app_1 | 2022/02/07 14:52:33 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:52:33 [INFO] Starting process for LV
app_1 | 2022/02/07 14:52:35 [INFO] Scraping victims
app_1 | 2022/02/07 14:52:37 [INFO] There are 0 new victims
app_1 | 2022/02/07 14:52:37 [INFO] Identifying removed victims
app_1 | 2022/02/07 14:52:37 [INFO] There are 0 removed victims
app_1 | 2022/02/07 14:52:37 [INFO] Finished LV
app_1 | 2022/02/07 14:52:37 [INFO] Starting process for Marketo
app_1 | 2022/02/07 14:52:37 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:52:37 [INFO] Starting process for LockData
app_1 | 2022/02/07 14:52:40 [INFO] Scraping victims
app_1 | 2022/02/07 14:52:42 [INFO] There are 0 new victims
app_1 | 2022/02/07 14:52:42 [INFO] Identifying removed victims
app_1 | 2022/02/07 14:52:42 [INFO] There are 0 removed victims
app_1 | 2022/02/07 14:52:42 [INFO] Finished LockData
app_1 | 2022/02/07 14:52:42 [INFO] Starting process for Rook
app_1 | 2022/02/07 14:52:42 [WARNING] No URL found in config for this actor, skipping
app_1 | 2022/02/07 14:52:42 [INFO] Finished all sites, exiting

Environment

• OS: [Ubuntu 20.04.3]
• How you are running it: [Docker with cronjob]

[captainGeech42]

This is likely due to scrapers being out of date on the sites you have configured. This tool isn't being actively developed, and is reliant on pull requests to update the scrapers. Sorry about that.

If you do update the scrapers and can open a pull request, I'd be happy to get that merged in.