Scrapy logo

Scrapy

A fast high-level web crawling & scraping framework for Python
244
239
+ 1
0

What is Scrapy?

It is the most popular web scraping framework in Python. An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.
Scrapy is a tool in the Web Scraping API category of a tech stack.
Scrapy is an open source tool with 50.4K GitHub stars and 10.3K GitHub forks. Here’s a link to Scrapy's open source repository on GitHub

Who uses Scrapy?

Companies
29 companies reportedly use Scrapy in their tech stacks, including WISESIGHT, Yintrust, and Witei.

Developers
204 developers on StackShare have stated that they use Scrapy.

Scrapy Integrations

Decisions about Scrapy

Here are some stack decisions, common use cases and reviews by companies and developers who chose Scrapy in their tech stack.

Howie Zhao
Full Stack Engineer at yintrust · | 2 upvotes · 12.8K views
Shared insights
on
ScrapyScrapy
at

There is no doubt that Scrapy is the preferred Python crawler framework.

See more

Scrapy Alternatives & Comparisons

What are some alternatives to Scrapy?
Selenium
Selenium automates browsers. That's it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well.
import.io
import.io is a free web-based platform that puts the power of the machine readable web in your hands. Using our tools you can create an API or crawl an entire website in a fraction of the time of traditional methods, no coding required.
BeautifulSoup
It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work.
Puppeteer
Puppeteer is a Node library which provides a high-level API to control headless Chrome over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome.
Apify
Apify is a platform that enables developers to create, customize and run cloud-based programs called actors that can, among other things, be used to extract data from any website using a few lines of JavaScript.
See all alternatives

Scrapy's Followers
239 developers follow Scrapy to keep up with related blogs and decisions.