What is Scrapy?
It is the most popular web scraping framework in Python. An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.
Scrapy is a tool in the Web Scraping API category of a tech stack.
Scrapy is an open source tool with 53.4K GitHub stars and 10.6K GitHub forks. Here’s a link to Scrapy's open source repository on GitHub
Who uses Scrapy?
Companies
29 companies reportedly use Scrapy in their tech stacks, including WISESIGHT, Yintrust, and Witei.
Developers
204 developers on StackShare have stated that they use Scrapy.
Scrapy Integrations
Decisions about Scrapy
Here are some stack decisions, common use cases and reviews by companies and developers who chose Scrapy in their tech stack.
Howie Zhao
Full Stack Engineer at yintrust · | 2 upvotes · 13.9K views
There is no doubt that Scrapy is the preferred Python crawler framework.
Scrapy Alternatives & Comparisons
What are some alternatives to Scrapy?
Selenium
Selenium automates browsers. That's it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well.
import.io
import.io is a free web-based platform that puts the power of the machine readable web in your hands. Using our tools you can create an API or crawl an entire website in a fraction of the time of traditional methods, no coding required.
BeautifulSoup
It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work.
Puppeteer
Puppeteer is a Node library which provides a high-level API to control headless Chrome over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome.
Postman
It is the only complete API development environment, used by nearly five million developers and more than 100,000 companies worldwide.
