firecrawl/firecrawl

🔥 The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data

⭐ 85745 | 🍴 6158

unclecode/crawl4ai

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN

⭐ 61003 | 🍴 6235

scrapy/scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

⭐ 59888 | 🍴 11257

NanmiCoder/MediaCrawler

小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频 ｜ 评论爬虫、微博帖子 ｜ 评论爬虫、百度贴吧帖子 ｜ 百度贴吧评论回复爬虫 | 知乎问答文章｜评论爬虫

⭐ 44294 | 🍴 9718