Skip to content

Blocking Bots

Jump to bottom
bitslip6 edited this page Feb 8, 2021 · 3 revisions

Preventing Bot Access

TL;DR

  • First require that all requests use your actual domain name. This will prevent requests directly to your IP address and stops the majority of Internet worms. Set "check_domain" to "block" and add your domain name to the valid_domains[] list like so (no need to include www) you should add as many domains as you are protecting with BitFire:
check_domain = "block"
valid_domains[] = "your_domain_name.com"
  • Next, enable only whitelisted domains. BitFire comes with almost 3 dozen pre-configured and verified bots from search engines and popular social media sites. Any bot impersonating approved bots on the list will be verified as authentic by checking the requests network origin and blocked if they fail.
whitelist_enable = "block"
  • Finally, ensure that browsers are really browsers. For User-Agents claiming to be web browsers, BitFire can send down a JavaScript challenge that the client must answer correctly before being allowed access. This takes < 100 milliseconds on average for the first page view and will drastically cut down on your bot activity.
require_full_browser = "block"

Intro

Most spam / security compromises happen from bots. Programs hacking programs. The Internet is a big place, and if you are trying to compromise as many sites as possible with a new 0-day attack, or spam links to "dating" sites, you are going to need to cover a lot of ground. The only way to do that effectively is with automated bots. BitFire has the best automated bot prevention available today for small / medium websites and businesses. There are 2 main types: bots pretending to be google/bing to bypass filtering and bots pretending to be users (Chrome/Firefox/Edge/Safari) - we cover both.

  • The first recommended feature to enable to "check_domain". This will require that all people visiting your site use the domain name and not the IP address. This will stop many worms and bots that search the internet looking for insecure CMS plugins (looking at you WordPress). After enabling this feature your website will no longer be available by going directly to the IP address. Only domain names listed in the config.ini will be allowed.

  • Next, decide if you want to whitelist or blacklist bots. Whitelist bots will only allow bots that you configure in config.ini. Blacklist bots will block any bots from the blacklist file (contains a list of about 100-200 known bad bots). Blacklisting is completely safe and will stop some obvious non-sense, but it is easy to bypass and unlikely to stop a real hack. If possible we highly recommend using the whitelist. If you are unsure if this may break your setup, set whitelist_enable = "report" and monitor the /tmp/bitfire.report.json file to see what types of traffic would be blocked. After you are satisfied that you will be allowing only good traffic, set whitelist_enable = "block"

  • Lastly, enable JavaScript verification of actual browsers. This will send a JavaScript math challenge to all browsers that must be solved correctly before they are allowed to access the website. When this feature is combined with whitelist_enable all automated attacks will be blocked. This will require any automated system (spam bot, Internet worm, exploit script, web scanner, etc) to be blocked. This setting is highly recomended.

** Notes **

Whitelisting.

  • To add a bot to the whitelist, take a unique part of the user-agent (like "curl", "wget", "googlebot", etc) and add it to the list of whitelist in config.ini. The value of the entry can be a regular expression matching the allowed source IP addresses, a regular expression matching the reverse domain lookup, or an AS number (prefixed with AS). Here are some examples:
; allow 'curl' from all local networks
botwhitelist[curl] = "192.168.*, 10.*, 127.0.0.1"

; allow 'custom agent' from mycustomagent.com
botwhitelist[custom agent] = "mycustomagent.com"

; allow 'some agent' from AS12312
botwhitelist[some agent] = "AS12312"

When enabling bot filtering, some services like, seochecker.pro, neilpatiel.com, accunetix.com, etc will not be able to connect to your web site. To easily add access for them, go to your dashboard and click on "bots" in the navigation. Then click "generate new link". A new link with an access token will be created. Copy and paste the created into your tool giving the remote system access.

Clone this wiki locally