This project investigates user engagement bias within the Indian news media, specifically focusing on The Times of India (TOI). As media consumption significantly impacts public opinion, this analysis aims to identify patterns in user interactions that align with specific political, social, or communal ideologies. By understanding these trends, we aspire to promote media literacy and accountability, contributing to a more balanced media landscape in India.
- Systematic web crawling and data scraping of TOI articles and user comments.
- Analysis of user engagement patterns to identify potential biases.
- Scalable data processing through parallel execution and chunked scraping.
- Selenium: For automating browser interactions and handling dynamic web content.
- Scrapy: For efficient web scraping and data extraction.
- Jupyter Notebook: For exploratory data analysis and prototyping.
- Python: The primary programming language used for development.
- Pickle: For data storage and serialization.
The data collection process involved:
- Category Links Extraction: Extracting links to various categories on the TOI homepage.
- Article Links Collection: Gathering article links from different categories, resulting in a total of 27,108 articles.
- Data Crawling: Extracting relevant data from each article, including user comments and engagement metrics.
To run the scripts, ensure you have the required dependencies installed. You can execute the scripts by running the main Python file or individual Jupyter Notebooks for specific tasks.
- Python 3.x
- Required libraries (install via
pip install -r requirements.txt)