s0md3v/Photon

• Platinum III

Python ProxyPool for web spider

• 1,672−16 Elo, change today
• 23.7k stars
• 5.4k forks
• 5mo ago

git clone https://github.com/jhao104/proxy_pool.git

• crawler
• http
• proxy
• redis

• Gold IV

A Powerful Spider(Web Crawler) System in Python.

• 1,538−8 Elo, change today
• 16.7k stars
• 3.6k forks
• 2y ago
• archived

git clone https://github.com/binux/pyspider.git

• crawler
• python

• Platinum IV

Elegant Scraper and Crawler Framework for Golang

• 1,658+9 Elo, change today
• 25.5k stars
• 1.8k forks
• 1mo ago

git clone https://github.com/gocolly/colly.git

• crawler
• crawling
• framework
• go

• Copper II

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

• 1,339−2 Elo, change today
• 6.7k stars
• 863 forks
• 5mo ago

git clone https://github.com/bda-research/node-crawler.git

• cheerio
• crawler
• extract-data
• javascript

• Bronze IV

A collection of awesome web crawler,spider in different languages

• 1,384−2 Elo, change today
• 7.3k stars
• 751 forks
• 2y ago

git clone https://github.com/BruceDone/awesome-crawler.git

• awesome
• crawler
• node-crawler
• scraper

• Bronze IV

越来越多的网站具有反爬虫特性，有的用图片隐藏关键数据，有的使用反人类的验证码，建立反反爬虫的代码仓库，通过与不同特性的网站做斗争（无恶意）提高技术。（欢迎提交难以采集的网站）（因工作原因，项目暂停）

• 1,386−1 Elo, change today
• 7.2k stars
• 2.1k forks
• 5y ago

git clone https://github.com/luyishisi/Anti-Anti-Spider.git

• geek
• python
• spider