Get Advice Icon

Need advice about which tool to choose?Ask the StackShare community!

Scrapy
Scrapy

81
59
+ 1
0
Selenium
Selenium

5.5K
4.1K
+ 1
461
Add tool

Scrapy vs Selenium: What are the differences?

What is Scrapy? A fast high-level web crawling & scraping framework for Python. It is the most popular web scraping framework in Python. An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

What is Selenium? Web Browser Automation. Selenium automates browsers. That's it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well.

Scrapy and Selenium are primarily classified as "Web Scraping API" and "Browser Testing" tools respectively.

Scrapy and Selenium are both open source tools. It seems that Scrapy with 33.5K GitHub stars and 7.87K forks on GitHub has more adoption than Selenium with 14.9K GitHub stars and 4.97K GitHub forks.

According to the StackShare community, Selenium has a broader approval, being mentioned in 777 company stacks & 443 developers stacks; compared to Scrapy, which is listed in 11 company stacks and 5 developer stacks.

What is Scrapy?

It is the most popular web scraping framework in Python. An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

What is Selenium?

Selenium automates browsers. That's it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well.
Get Advice Icon

Need advice about which tool to choose?Ask the StackShare community!

Why do developers choose Scrapy?
Why do developers choose Selenium?
    Be the first to leave a pro

    Sign up to add, upvote and see more prosMake informed product decisions

      Be the first to leave a con
        Be the first to leave a con
        What companies use Scrapy?
        What companies use Selenium?

        Sign up to get full access to all the companiesMake informed product decisions

        What tools integrate with Scrapy?
        What tools integrate with Selenium?

        Sign up to get full access to all the tool integrationsMake informed product decisions

        What are some alternatives to Scrapy and Selenium?
        import.io
        import.io is a free web-based platform that puts the power of the machine readable web in your hands. Using our tools you can create an API or crawl an entire website in a fraction of the time of traditional methods, no coding required.
        BeautifulSoup
        It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work.
        Puppeteer
        Puppeteer is a Node library which provides a high-level API to control headless Chrome over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome.
        Portia
        Portia is an open source tool that lets you get data from websites. It facilitates and automates the process of data extraction. This visual web scraper works straight from your browser, so you don't need to download or install anything.
        Kimono
        You don't need to write any code or install any software to extract data with Kimono. The easiest way to use Kimono is to add our bookmarklet to your browser's bookmark bar. Then go to the website you want to get data from and click the bookmarklet. Select the data you want and Kimono does the rest. We take care of hosting the APIs that you build with Kimono and running them on the schedule you specify. Use the API output in JSON or as CSV files that you can easily paste into a spreadsheet.
        See all alternatives
        Decisions about Scrapy and Selenium
        Benjamin Poon
        Benjamin Poon
        QA Manager - Engineering at HBC Digital ยท | 8 upvotes ยท 223.3K views
        Nightwatchjs
        Nightwatchjs
        Selenium
        Selenium
        JavaScript
        JavaScript
        Cucumber
        Cucumber
        GitHub
        GitHub
        GoCD
        GoCD
        Docker
        Docker
        ExpressJS
        ExpressJS
        React
        React
        PostgreSQL
        PostgreSQL

        For our digital QA organization to support a complex hybrid monolith/microservice architecture, our team took on the lofty goal of building out a commonized UI test automation framework. One of the primary requisites included a technical minimalist threshold such that an engineer or analyst with fundamental knowledge of JavaScript could automate their tests with greater ease. Just to list a few: - Nightwatchjs - Selenium - Cucumber - GitHub - Go.CD - Docker - ExpressJS - React - PostgreSQL

        With this structure, we're able to combine the automation efforts of each team member into a centralized repository while also providing new relevant metrics to business owners.

        See more
        Kevin Roulleau
        Kevin Roulleau
        QA Engineer Freelance at happn ยท | 5 upvotes ยท 66.2K views
        athappnhappn
        WebdriverIO
        WebdriverIO
        Appium
        Appium
        Selenium
        Selenium

        I chose WebdriverIO and Appium to implement a E2E tests solution on a native mobile app. WebdriverIO goes well beyond just implementing the Selenium / Appium protocol and allows to run tests in parallel out of the box. Appium has the big advantage of supporting iOS and Android platforms, so the test codebase and tools are exactly the same, which greatly reduces the learning curve and implementation time.

        See more
        Interest over time
        Reviews of Scrapy and Selenium
        Review ofSeleniumSelenium

        we at www.cloudqa.io use selenium hub and node architecture to execute our test cases its a very powerful tool to automate web applications.

        How developers use Scrapy and Selenium
        Avatar of Kang Hyeon Ku
        Kang Hyeon Ku uses SeleniumSelenium

        ๊ฐ€์žฅ ๊ฐ•๋ ฅํ•œ ์›น UI / UX ํ…Œ์ŠคํŠธ ํˆด ์ด์ง€๋งŒ, ๋ฌธ์ œ๋„ ๋งŽ๋‹ค. iframe ์•ˆ์— ์žˆ๋Š” ๊ฑด ์ž˜ select ๊ฐ€ ์•ˆ๋˜๊ธฐ๋„ ํ•˜๊ณ , ํ™”๋ฉด์ด ์ž‘์•„ ์ปดํฌ๋„ŒํŠธ๊ฐ€ ํ™”๋ฉด์— ๋ณด์ด์ง€ ์•Š์œผ๋ฉด ๋˜ select ํ•  ์ˆ˜ ์—†๋‹ค. ํ•˜์ง€๋งŒ ๋ธŒ๋ผ์šฐ์ €๋ฅผ ์žˆ๋Š” ๊ทธ๋Œ€๋กœ ์ œ์–ด ํ•  ์ˆ˜ ์žˆ๋‹ค๋Š” ๊ฒŒ ๊ฐ€์žฅ ํฐ ๋งค๋ ฅ์ด๋‹ค.

        ps. selenide ์™€ ๊ฐ™์ด ์“ฐ๋ฉด ์ฝ”๋“œ๊ฐ€ ๋งŽ์ด ๊ฐœ์„ ๋œ๋‹ค.

        Avatar of samdfonseca
        samdfonseca uses SeleniumSelenium

        Maintaining and expanding automated regression tests is one of my primary responsibilities at CrowdTwist. We use Selenium to test the front-end parts of our platform.

        Avatar of Ralic Lo
        Ralic Lo uses SeleniumSelenium

        Used Selenium web-driver for automated QA testing in continuous integration and continuous deployment.

        Avatar of Owen Miller
        Owen Miller uses SeleniumSelenium
        • Navigating our site for tests
        • Navigating other sites that discriminate against non-human visitors
        Avatar of Ataccama
        Ataccama uses SeleniumSelenium

        Selenium is a great way to automatically test if a build is sane and can be deployed.

        How much does Scrapy cost?
        How much does Selenium cost?
        Pricing unavailable
        Pricing unavailable
        News about Scrapy
        More news