Need advice about which tool to choose?Ask the StackShare community!

Scrapy

177
174
+ 1
0
Selenium

10.7K
8.2K
+ 1
519
Add tool

Scrapy vs Selenium: What are the differences?

What is Scrapy? A fast high-level web crawling & scraping framework for Python. It is the most popular web scraping framework in Python. An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

What is Selenium? Web Browser Automation. Selenium automates browsers. That's it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well.

Scrapy and Selenium are primarily classified as "Web Scraping API" and "Browser Testing" tools respectively.

Scrapy and Selenium are both open source tools. It seems that Scrapy with 33.5K GitHub stars and 7.87K forks on GitHub has more adoption than Selenium with 14.9K GitHub stars and 4.97K GitHub forks.

According to the StackShare community, Selenium has a broader approval, being mentioned in 777 company stacks & 443 developers stacks; compared to Scrapy, which is listed in 11 company stacks and 5 developer stacks.

Advice on Scrapy and Selenium
Needs advice
on
WebdriverIO
Selenium
and
Jest
in

we are having one web application developed in Reacts.js. in the application, we have only 4 to 5 pages that we need to test. I am having experience in selenium with java. Please suggets which tool I should use. and why ............................ ............................ .............................

See more
Replies (1)
Prashanth Marappa
Senior Software Engineer at Mphasis · | 1 upvotes · 91.9K views
Recommends
Selenium

with the help of selenium we can automate react js for functional testing

See more
Get Advice from developers at your company using Private StackShare. Sign up for Private StackShare.
Learn More
Pros of Scrapy
Pros of Selenium
    Be the first to leave a pro
    • 169
      Automates browsers
    • 154
      Testing
    • 101
      Essential tool for running test automation
    • 24
      Record-Playback
    • 24
      Remote Control
    • 8
      Data crawling
    • 7
      Supports end to end testing
    • 6
      Functional testing
    • 6
      Easy set up
    • 4
      The Most flexible monitoring system
    • 3
      Easy to integrate with build tools
    • 3
      End to End Testing
    • 2
      Integration Tests
    • 2
      Comparing the performance selenium is faster than jasm
    • 2
      Record and playback
    • 2
      Compatible with Python
    • 2
      Easy to scale
    • 0
      Integrated into Selenium-Jupiter framework

    Sign up to add or upvote prosMake informed product decisions

    Cons of Scrapy
    Cons of Selenium
      Be the first to leave a con
      • 7
        Flaky tests
      • 2
        Slow as needs to make browser (even with no gui)

      Sign up to add or upvote consMake informed product decisions

      What is Scrapy?

      It is the most popular web scraping framework in Python. An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

      What is Selenium?

      Selenium automates browsers. That's it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well.

      Need advice about which tool to choose?Ask the StackShare community!

      What companies use Scrapy?
      What companies use Selenium?
      See which teams inside your own company are using Scrapy or Selenium.
      Sign up for Private StackShareLearn More

      Sign up to get full access to all the companiesMake informed product decisions

      What tools integrate with Scrapy?
      What tools integrate with Selenium?

      Sign up to get full access to all the tool integrationsMake informed product decisions

      Blog Posts

      What are some alternatives to Scrapy and Selenium?
      import.io
      import.io is a free web-based platform that puts the power of the machine readable web in your hands. Using our tools you can create an API or crawl an entire website in a fraction of the time of traditional methods, no coding required.
      BeautifulSoup
      It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work.
      Puppeteer
      Puppeteer is a Node library which provides a high-level API to control headless Chrome over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome.
      ParseHub
      Web Scraping and Data Extraction ParseHub is a free and powerful web scraping tool. With our advanced web scraper, extracting data is as easy as clicking on the data you need. ParseHub lets you turn any website into a spreadsheet or API w
      Portia
      Portia is an open source tool that lets you get data from websites. It facilitates and automates the process of data extraction. This visual web scraper works straight from your browser, so you don't need to download or install anything.
      See all alternatives