unclecode/crawl4ai

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN

⭐ 61213 | 🍴 6258

scrapy/scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

⭐ 60455 | 🍴 11327

NanmiCoder/MediaCrawler

小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频 ｜ 评论爬虫、微博帖子 ｜ 评论爬虫、百度贴吧帖子 ｜ 百度贴吧评论回复爬虫 | 知乎问答文章｜评论爬虫

⭐ 44670 | 🍴 9768

BuilderIO/gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL

⭐ 22180 | 🍴 2388