Overcoming Social Media API Restrictions: Building an Effective Web Scraper
Loading...
Authors
Harrell, Nicholas
Cruickshank, Iain
Master, Alexander
Issue Date
2024
Type
Article
Language
en
Keywords
Alternative Title
Abstract
As social media platform application programming interfaces (APIs) are becoming more restrictive and costly to use, there is a considerable risk that researchers will be unable to address research problems related to online discourse. This paper presents a detailed examination of an effective approach to scraping X (formerly Twitter) data, leveraging the Selenium WebDriver for automated interaction with web pages. This technique circumvents the limitations of X’s dynamic content generation and JavaScript-dependent interface, providing a robust alternative to traditional API-based data retrieval methods. By emulating human navigation patterns, this method offers insights into extracting real-time social media data, including tweets, likes, and retweets, which are crucial for various analytical applications.
Description
Citation
Harrell, Nicholas, Iain Cruickshank, and Alexander Master. 2024. “Overcoming Social Media API Restrictions: Building an Effective Web Scraper.” Paper presented at ICWSM 2024, United States. Workshop Proceedings of the 18th International AAAI Conference on Web and Social Media, June 1. https://doi.org/10.36190/2024.72.
Publisher
Workshop Proceedings of the 18th International AAAI Conference on Web and Social Media