Scrapy logo

Scrapy

A fast high-level web crawling & scraping framework for Python
168
166
+ 1
0

What is Scrapy?

It is the most popular web scraping framework in Python. An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.
Scrapy is a tool in the Web Scraping API category of a tech stack.
Scrapy is an open source tool with 41K GitHub stars and 9.2K GitHub forks. Here’s a link to Scrapy's open source repository on GitHub

Who uses Scrapy?

Companies
23 companies reportedly use Scrapy in their tech stacks, including WISESIGHT, Yintrust, and Witei.

Developers
143 developers on StackShare have stated that they use Scrapy.

Scrapy Integrations

Decisions about Scrapy

Here are some stack decisions, common use cases and reviews by companies and developers who chose Scrapy in their tech stack.

Howie Zhao
Full Stack Engineer at yintrust · | 1 upvote · 2.2K views
Shared insights
on
Scrapy
at

There is no doubt that Scrapy is the preferred Python crawler framework.

See more

Scrapy Alternatives & Comparisons

What are some alternatives to Scrapy?
Selenium
Selenium automates browsers. That's it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well.
import.io
import.io is a free web-based platform that puts the power of the machine readable web in your hands. Using our tools you can create an API or crawl an entire website in a fraction of the time of traditional methods, no coding required.
BeautifulSoup
It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work.
Puppeteer
Puppeteer is a Node library which provides a high-level API to control headless Chrome over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome.
ParseHub
Web Scraping and Data Extraction ParseHub is a free and powerful web scraping tool. With our advanced web scraper, extracting data is as easy as clicking on the data you need. ParseHub lets you turn any website into a spreadsheet or API w
See all alternatives

Scrapy's Followers
166 developers follow Scrapy to keep up with related blogs and decisions.