bestofgithub.devRepositories with the GitHub topic “web-scraping”: 35 repositories with 1M stars between them, mostly written in Python, TypeScript and JavaScript. The highest-rated is D4Vinci/Scrapling, #24 on the bestofgithub Elo ladder.
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place
go install github.com/gosom/google-maps-scraper@latestTurn any website into a structured API. Extract, automate, search and monitor the web.
git clone https://github.com/getmaxun/maxun && cd maxun && docker compose up -dPython binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
git clone https://github.com/lexiforest/curl_cffi.gitOfficial Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse, crawl, map, Agent and Monitor, from any MCP client.
git clone https://github.com/firecrawl/firecrawl-mcp-server.gitBest headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
cargo install --git https://github.com/lexmount/moliThis GitHub repo is a powerhouse collection of APIs you can start using immediately to build everything from simple automations to full-scale applications. One of the most valuable API lists on GitHub—period. 💪
git clone https://github.com/cporter202/API-mega-list.git🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev
pipx install scraplingSupercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
git clone https://github.com/firecrawl/firecrawl && cd firecrawl && docker compose up -d🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy…
Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.
Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a…
Clone any website with one command using AI coding agents
Scrapy, a fast high-level web crawling & scraping framework for Python.
Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for…
Python scraper based on AI
scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews…
Stealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint…
🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili…
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and…
The SDK to extract data and interact with any site on the web. Get started with Claude Code, Codex, Eve, Mastra, and…
Turn any website into a structured API. Extract, automate, search and monitor the web.
Air-gapped, privacy-focused, open-source NotebookLM alternative. AI agent that researches, transforms, and edits your…
Stealth headless browser for AI agents — bypass Cloudflare, bot detection, and anti-scraping. Drop-in…
Open-source AI agent for web automation. Use your real, logged-in browser to automate daily tasks.
This GitHub repo is a powerhouse collection of APIs you can start using immediately to build everything from simple…
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
📊 Blazing fast Python framework for web crawling, testing, scraping, and stealth. Supports pytest. Bypasses…
Official Firecrawl MCP server, the context API for AI agents: search, scrape, and interact with the web, plus parse…
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON…
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data…
High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and…
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2…
Automated CAPTCHA solver for your browser. Works with Selenium, Puppeteer, Playwright, and more.
jsoup: the Java HTML parser, built for HTML editing, cleaning, scraping, and XSS safety.
List of libraries, tools and APIs for web scraping and data processing.
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
Browser automation CLI built for AI agents. Break through anti-bot walls, hand off to humans across platforms when…
Lighter web automation with Python
A Chrome DevTools Protocol driver for web automation and scraping.
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.
This is a lightweight, headless browser purpose-built for AI agents (such as Codex or Claude Code) to perform browser…