scrapy/scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

⭐ 60518 | 🍴 11332

BuilderIO/gpt-crawler

Crawl a site to generate knowledge files to create your own custom GPT from a URL

⭐ 22185 | 🍴 2387

D4Vinci/Scrapling

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

⭐ 21050 | 🍴 1469

projectdiscovery/katana

A next-generation crawling and spidering framework.

⭐ 15631 | 🍴 957