bestofgithub.dev

← Back to the directory

apify/crawlee-python

View on GitHub

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.

apify/crawlee-python is #4,734 on the bestofgithub Elo ladder, in Gold V, with a rating of 1,532 (+32 this week). It has 9,675 stars, 20 gained in the last week.

Gold V

1,532 Elo

  • #4,734 on the ladder
  • +6 today · +32 this week
  • Peak 1,532
6 Elo to Gold IV

Stars

9,675

  • +2 today
  • +20 this week
  • – this month

Repository

Forks
833
Language
Python
Last push

Achievements

bestofgithub badgeMaintainer? Show your league in your README

Card: your league's emblem, from Copper to Legendary

[![bestofgithub](https://bestofgithub.dev/badge/apify/crawlee-python.svg?style=card)](https://bestofgithub.dev/repo/apify/crawlee-python)

Flat: fits beside your other badges

[![bestofgithub](https://bestofgithub.dev/badge/apify/crawlee-python.svg)](https://bestofgithub.dev/repo/apify/crawlee-python)

Install

git
git clone https://github.com/apify/crawlee-python.git

Similar repositories

apify/crawlee

Open on GitHubTypeScript

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

  • 1,821−2 Elo, change today
  • 26.4k stars
  • 1.7k forks
  • 135 watchers
git
git clone https://github.com/apify/crawlee.git
v3.18.21mo ago
  • apify
  • automation
  • crawler
  • crawling

unclecode/crawl4ai

Open on GitHubPython

Official Crawl4AI Cloud: a hosted web scraping, extraction, and search API for AI agents. Free API key, 1,000 credits a month, no card. Fully open source: this repo is the Apache-2.0 crawler that powers it, and you can self-host it.

  • 2,099−3 Elo, change today
  • 87.3k stars
  • 9.1k forks
  • 422 watchers
pipx install Crawl4AI
v0.9.51mo ago
  • ai
  • ai-agents
  • crawl4ai-cloud
  • crawler

scrapy/scrapy

Open on GitHubPython

Scrapy, a fast high-level web crawling & scraping framework for Python.

  • 2,020+7 Elo, change today
  • 65.4k stars
  • 12.1k forks
  • 1.7k watchers
brew install scrapy
2.20.09d ago
  • crawler
  • crawling
  • framework
  • hacktoberfest

gocolly/colly

Open on GitHubGo

Elegant Scraper and Crawler Framework for Golang

  • 1,658+9 Elo, change today
  • 25.5k stars
  • 1.8k forks
git
git clone https://github.com/gocolly/colly.git
  • crawler
  • crawling
  • framework
  • go

alirezamika/autoscraper

Open on GitHubPython

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

  • 1,466+12 Elo, change today
  • 8k stars
  • 834 forks
git
git clone https://github.com/alirezamika/autoscraper.git
  • ai
  • artificial-intelligence
  • automation
  • crawler

codelucas/newspaper

Open on GitHubPython

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

  • 1,513−4 Elo, change today
  • 15.1k stars
  • 2.1k forks
git
git clone https://github.com/codelucas/newspaper.git
  • crawler
  • crawling
  • news
  • news-aggregator

README

Loading README…