twitter-stream-bot-data-gatherer

An application to watch the Twitter stream and send accounts to the Botometer API for analysis. The results are stored in a SQLite database.

Prerequisites

Before using this application, credentials for the following are required:

• The Twitter API

• The Botometer API

Twitter API Credentials

The application connects to the Twitter stream API to query Twitter user's to Botometer for analysis.

To get access to the Twitter API, a developer account is needed. You can apply for one here and find more information about the application process here.

Once you have a developer account, go to the developer portal to create an application.

When creating the application, provide a unique name and follow the instructions. The application's API key and secret will be shown (aka. consumer_key and consumer_secret). Furthermore, you will need to generate an access token and access token secret to use the Twitter stream API. Go to your application, click Keys and tokens > Access Token and Secret > Generate. Store the credentials somewhere safe as they will be needed later.

Botometer API Credentials

The application uses the Botometer API to collect bot scores for Twitter users.

To get access to the Botometer API, a RapidAPI account is needed. You can create one here.

Once you have a RapidAPI account, go to the Botometer API pricing page and subscribe to the Basic or Pro plan.

Note

The Pro plan requires a credit or debit card to be added to the RapidAPI account.

Once you have chosen a plan, go to the RapidAPI developer dashboard. Then, go to security page for the default-application. The application key (API key) will be shown. Store it somewhere safe as it will be needed later.

Further details on accessing the Botometer API can be found here.

Dependencies

Install the application's dependencies using one of the options below:

# Poetry
poetry install --no-dev
# Pip
pip install -r requirements.txt
# Or use Docker!
# Please refer to the heading below

Usage

$ python twitter-stream-bot-data-gatherer/main.py -h

usage: main.py [-h] [-t TRACK] [-f DATABASE_NAME] [-d]
               rapidapi_key twitter_app_auth

An application to watch the Twitter stream and send accounts to the Botometer
API for analysis. The results are stored in a SQLite database.

positional arguments:
  rapidapi_key          Botometer Rapid API key.
  twitter_app_auth      Twitter application credentials.

optional arguments:
  -h, --help            show this help message and exit
  -t TRACK, --track TRACK
                        A hashtag to track. Can be specified more than once.
  -f DATABASE_NAME, --database_name DATABASE_NAME
                        Name of the database file. Defaults to: twitter-
                        stream-bot-data-gatherer.
  -d, --debug           Enable debug messages.

Tracking a Single Hashtag

python twitter-stream-bot-data-gatherer/main.py <rapidapi_key> \
  '{"consumer_key": "", "consumer_secret": "", "access_token": "", "access_token_secret": ""}' \
  --track '#StandWithUkriane'

Tracking Multiple Hashtags

python twitter-stream-bot-data-gatherer/main.py <rapidapi_key> \
  '{"consumer_key": "", "consumer_secret": "", "access_token": "", "access_token_secret": ""}' \
  -t '#StandWithUkriane' \
  -t '#RefugeesWelcome' \
  -t '#Ukrania'

Modifying the SQLite Database Name

python twitter-stream-bot-data-gatherer/main.py <rapidapi_key> \
  '{"consumer_key": "", "consumer_secret": "", "access_token": "", "access_token_secret": ""}' \
  -t '#StandWithUkriane' \
  # The database will be named 'twitter-db.db'
  -f 'twitter-db'

Docker 🐋

Build the container image:

docker build -t twitter-stream-bot-data-gatherer:v0.1.0 .

Run the application and persist the SQLite database:

docker run -it --name twitter-stream-bot-data-gatherer \
    -v absolute/path/to/store/db:/usr/src/app/db \
    twitter-stream-bot-data-gatherer:v0.1.0 <rapidapi_key> \
    '{"consumer_key": "", "consumer_secret": "", "access_token": "", "access_token_secret": ""}' \
    --track '#StandWithUkriane'

SQLite Database Structure

The SQLite database will contain one table called data. Within the table will be three columns:

1. screen_name: The screen name of the Twitter account.

2. status_json: The full JSON response of the Tweet Object (aka. Status) from the Twitter stream.

3. botometer_json: The full JSON response from the Botometer API.

My Related Work

• dbrennand/Final-Year-Project - An application that generates a report to identify potential bots that a Twitter user is following.

Acknowledgments

This application would not be possible without the work of the Observatory on Social Media at Indiana University who created Botometer. Please see the Botometer FAQ page for further information on Botometer.

Authors -- Contributors

• dbrennand - Author

License

This project is licensed under the MIT License - see the LICENSE for details.