scrapy/scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

⭐ 60506 | 🍴 11332

BuilderIO/gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL

⭐ 22183 | 🍴 2388

D4Vinci/Scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

⭐ 20451 | 🍴 1414

projectdiscovery/katana

A next-generation crawling and spidering framework.

⭐ 15625 | 🍴 957