Top Crawler repositories

Biggest movers this week

git clone https://github.com/getmaxun/maxun && cd maxun && docker compose up -d

pipx install jmcomic

git clone https://github.com/alirezamika/autoscraper.git

git clone https://github.com/projectdiscovery/katana.git

git clone https://github.com/adbar/trafilatura.git

git clone https://github.com/apify/crawlee-python.git

Trending today

pipx install scrapling

git clone https://github.com/firecrawl/firecrawl && cd firecrawl && docker compose up -d

Top 32 on the Elo ladder

1. #24
   D4Vinci/Scrapling

   🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy…

   • Legendary #24

   2,448 Elo +11103kPython
2. #46
   firecrawl/firecrawl

   Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

   • Legendary #46

   2,347 Elo −11202kTypeScript
3. #242
   unclecode/crawl4ai

   Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a…

   • Diamond II

   2,099 Elo −1987.3kPython
4. #390
   scrapy/scrapy

   Scrapy, a fast high-level web crawling & scraping framework for Python.

   • Diamond III

   2,020 Elo +765.4kPython
5. #700
   ScrapeGraphAI/Scrapegraph-ai

   Python scraper based on AI

   • Diamond V

   1,916 Elo −1432.7kPython
6. #984
   Evil0ctal/Douyin_TikTok_Download_API

   🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API，支持 AI agent 通过 MCP 调用与离线语音转文字，Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili…

   • Emerald II

   1,859 Elo −2921.4kPython
7. #1,187
   hect0x7/JMComic-Crawler-Python

   Python API for JMComic | 提供Python API访问禁漫天堂，同时支持网页端和移动端 | 禁漫天堂GitHub Actions下载器🚀

   • Emerald III

   1,829 Elo +618kPython
8. #1,245
   apify/crawlee

   Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and…

   • Emerald III

   1,821 Elo +1626.4kTypeScript
9. #1,315
   getmaxun/maxun

   Turn any website into a structured API. Extract, automate, search and monitor the web.

   • Emerald III

   1,810 Elo +11718.4kTypeScript
10. #1,629
    NaiboWang/EasySpider

    A visual no-code/code-free web…

    • Emerald V

    1,769 Elo −944.7kJavaScript
11. #2,157
    iawia002/lux

    👾 Fast and simple video download library and CLI tool written in Go

    • Platinum II

    1,710 Elo −4131.8kGo
12. #2,509
    projectdiscovery/katana

    A next-generation crawling and spidering framework.

    • Platinum III

    1,675 Elo +4817.8kGo
13. #2,541
    jhao104/proxy_pool

    Python ProxyPool for web spider

    • Platinum III

    1,672 Elo −1923.7kPython
14. #2,716
    gocolly/colly

    Elegant Scraper and Crawler Framework for Golang

    • Platinum IV

    1,658 Elo +725.5kGo
15. #4,009
    s0md3v/Photon

    Incredibly fast crawler designed for OSINT.

    • Gold III

    1,573 Elo −713.3kPython
16. #4,408
    injetlee/Python

    Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机

    • Gold IV

    1,550 Elo −5310.9kPython
17. #4,632
    binux/pyspider

    A Powerful Spider(Web Crawler) System in Python.

    • Gold IV

    1,538 Elo −4116.7kPython
18. #4,713
    adbar/trafilatura

    Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON…

    • Gold V

    1,533 Elo +467.1kPython
19. #4,734
    apify/crawlee-python

    Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data…

    • Gold V

    1,532 Elo +329.6kPython
20. #4,875
    TeamWiseFlow/xiaobei

    为OPC/中小微企业量身打造的自媒体获客智能体

    • Gold V

    1,525 Elo +98.7kPython
21. #5,009
    shengqiangzhang/examples-of-web-crawlers

    一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to…

    • Silver I

    1,519 Elo −4014.7kHTML
22. #5,115
    crawlab-team/crawlab

    Distributed web crawler admin platform for spiders management regardless of languages and frameworks…

    • Silver I

    1,514 Elo −1812.3kGo
23. #5,129
    codelucas/newspaper

    newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

    • Silver I

    1,513 Elo −3415.1kPython
24. #5,302
    guyueyingmu/avbook

    AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links…

    • Silver I

    1,505 Elo −1510.1kPHP
25. #5,789
    code4craft/webmagic

    A scalable web crawler framework for Java.

    • Silver II

    1,483 Elo −4711.6kJava
26. #6,029
    lorien/awesome-web-scraping

    List of libraries, tools and APIs for web scraping and data processing.

    • Silver III

    1,472 Elo +18.2kJavaScript
27. #6,165
    alirezamika/autoscraper

    A Smart, Automatic, Fast and Lightweight Web Scraper for Python

    • Silver III

    1,466 Elo +508kPython
28. #6,325
    ssssssss-team/spider-flow

    新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

    • Silver IV

    1,459 Elo −2011.3kJava
29. #7,923
    mvdctop/Movie_Data_Capture

    The AI local media metadata crawler

    • Bronze III

    1,398 Elo −117.4kPython
30. #8,275
    BruceDone/awesome-crawler

    A collection of awesome web crawler,spider in different languages

    • Bronze IV

    1,384 Elo +77.3k
31. #8,326
    chyroc/WechatSogou

    基于搜狗微信搜索的微信公众号爬虫接口

    • Bronze IV

    1,383 Elo −36.4kPython
32. #9,193
    bda-research/node-crawler

    Web Crawler/Spider for NodeJS + server-side jQuery ;-)

    • Copper II

    1,339 Elo +86.7kTypeScript