bestofgithub.dev

← Back to the directory

apify/crawlee

View on GitHub

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

apify/crawlee is #1,245 on the bestofgithub Elo ladder, in Emerald III, with a rating of 1,821 (+16 this week). It has 26,453 stars, 73 gained in the last week.

Emerald III

1,821 Elo

  • #1,245 on the ladder
  • −2 today · +16 this week
  • Peak 1,823
18 Elo to Emerald II

Stars

26,453

  • +9 today
  • +73 this week
  • – this month

Repository

Forks
1.7k
Watchers
135
Language
TypeScript
Last push
Category
Web Frameworks

Achievements

bestofgithub badgeMaintainer? Show your league in your README

Card: your league's emblem, from Copper to Legendary

[![bestofgithub](https://bestofgithub.dev/badge/apify/crawlee.svg?style=card)](https://bestofgithub.dev/repo/apify/crawlee)

Flat: fits beside your other badges

[![bestofgithub](https://bestofgithub.dev/badge/apify/crawlee.svg)](https://bestofgithub.dev/repo/apify/crawlee)

Install

git
git clone https://github.com/apify/crawlee.git
v3.18.21mo ago

Similar repositories

apify/crawlee-python

Open on GitHubPython

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

  • 1,532+6 Elo, change today
  • 9.6k stars
  • 833 forks
git
git clone https://github.com/apify/crawlee-python.git
  • apify
  • automation
  • beautifulsoup
  • crawler

unclecode/crawl4ai

Open on GitHubPython

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

  • 2,099−3 Elo, change today
  • 87.3k stars
  • 9.1k forks
  • 422 watchers
pipx install Crawl4AI
v0.9.51mo ago
  • ai
  • ai-agents
  • crawl4ai-cloud
  • crawler

gocolly/colly

Open on GitHubGo

Elegant Scraper and Crawler Framework for Golang

  • 1,658+9 Elo, change today
  • 25.5k stars
  • 1.8k forks
git
git clone https://github.com/gocolly/colly.git
  • crawler
  • crawling
  • framework
  • go

scrapy/scrapy

Open on GitHubPython

Scrapy, a fast high-level web crawling & scraping framework for Python.

  • 2,020+7 Elo, change today
  • 65.4k stars
  • 12.1k forks
  • 1.7k watchers
brew install scrapy
2.20.09d ago
  • crawler
  • crawling
  • framework
  • hacktoberfest

lexmount/moli

Open on GitHubRust

Best headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rust

  • 1,985−3 Elo, change today
  • 23.8k stars
  • 679 forks
  • 50 watchers
cargo
cargo install --git https://github.com/lexmount/moli
v1.1.243d ago
  • ai-agents
  • ai-tools
  • browser
  • browser-automation

getmaxun/maxun

Open on GitHubTypeScript

Turn any website into a structured API. Extract, automate, search and monitor the web.

  • 1,810+29 Elo, change today
  • 18.4k stars
  • 1.5k forks
  • 85 watchers
docker
git clone https://github.com/getmaxun/maxun && cd maxun && docker compose up -d
v0.1.021d ago
  • agents
  • api
  • automation
  • browser-automation

README

Loading README…