bestofgithub.dev

← Back to the directory

BruceDone/awesome-crawler

View on GitHub

A collection of awesome web crawler,spider in different languages

BruceDone/awesome-crawler is #8,268 on the bestofgithub Elo ladder, in Bronze IV, with a rating of 1,384 (+7 this week). It has 7,370 stars, 6 gained in the last week.

Bronze IV

1,384 Elo

  • #8,268 on the ladder
  • −1 today · +7 this week
  • Peak 1,393
6 Elo to Bronze III

Stars

7,370

  • +1 today
  • +6 this week
  • – this month

Achievements

bestofgithub badgeMaintainer? Show your league in your README

Card: your league's emblem, from Copper to Legendary

[![bestofgithub](https://bestofgithub.dev/badge/BruceDone/awesome-crawler.svg?style=card)](https://bestofgithub.dev/repo/BruceDone/awesome-crawler)

Flat: fits beside your other badges

[![bestofgithub](https://bestofgithub.dev/badge/BruceDone/awesome-crawler.svg)](https://bestofgithub.dev/repo/BruceDone/awesome-crawler)

Install

git
git clone https://github.com/BruceDone/awesome-crawler.git

Similar repositories

gocolly/colly

Open on GitHubGo

Elegant Scraper and Crawler Framework for Golang

  • 1,658+8 Elo, change today
  • 25.5k stars
  • 1.8k forks
git
git clone https://github.com/gocolly/colly.git
  • crawler
  • crawling
  • framework
  • go

s0md3v/Photon

Open on GitHubPython

Incredibly fast crawler designed for OSINT.

  • 1,572−3 Elo, change today
  • 13.3k stars
  • 1.6k forks
git
git clone https://github.com/s0md3v/Photon.git
  • crawler
  • information-gathering
  • osint
  • python

firecrawl/firecrawl

Open on GitHubTypeScript

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

  • 2,347+1 Elo, change today
  • +241 stars gained
  • 202k stars
  • 10.5k forks
  • 500 watchers
docker
git clone https://github.com/firecrawl/firecrawl && cd firecrawl && docker compose up -d
v2.11.04mo ago
  • ai
  • ai-agents
  • ai-crawler
  • ai-scraping

binux/pyspider

Open on GitHubPython

A Powerful Spider(Web Crawler) System in Python.

  • 1,538−8 Elo, change today
  • 16.7k stars
  • 3.6k forks
  • archived
git
git clone https://github.com/binux/pyspider.git
  • crawler
  • python

unclecode/crawl4ai

Open on GitHubPython

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

  • 2,099−4 Elo, change today
  • 87.4k stars
  • 9.1k forks
  • 422 watchers
pipx install Crawl4AI
v0.9.51mo ago
  • ai
  • ai-agents
  • crawl4ai-cloud
  • crawler

README

Loading README…