Get Advice Icon

Need advice about which tool to choose?Ask the StackShare community!

Scrapy
Scrapy

42
17
+ 1
0
Selenium
Selenium

3.8K
2.4K
+ 1
444
Add tool

Scrapy vs Selenium: What are the differences?

What is Scrapy? A fast high-level web crawling & scraping framework for Python. It is the most popular web scraping framework in Python. An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

What is Selenium? Web Browser Automation. Selenium automates browsers. That's it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well.

Scrapy and Selenium are primarily classified as "Web Scraping API" and "Browser Testing" tools respectively.

Scrapy and Selenium are both open source tools. It seems that Scrapy with 33.5K GitHub stars and 7.87K forks on GitHub has more adoption than Selenium with 14.9K GitHub stars and 4.97K GitHub forks.

According to the StackShare community, Selenium has a broader approval, being mentioned in 777 company stacks & 443 developers stacks; compared to Scrapy, which is listed in 11 company stacks and 5 developer stacks.

What is Scrapy?

It is the most popular web scraping framework in Python. An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

What is Selenium?

Selenium automates browsers. That's it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well.
Get Advice Icon

Need advice about which tool to choose?Ask the StackShare community!

Why do developers choose Scrapy?
Why do developers choose Selenium?
    Be the first to leave a pro

    Sign up to add, upvote and see more prosMake informed product decisions

    What are the cons of using Scrapy?
    What are the cons of using Selenium?
      Be the first to leave a con
        Be the first to leave a con
        What companies use Scrapy?
        What companies use Selenium?

        Sign up to get full access to all the companiesMake informed product decisions

        What tools integrate with Scrapy?
        What tools integrate with Selenium?

        Sign up to get full access to all the tool integrationsMake informed product decisions

        What are some alternatives to Scrapy and Selenium?
        import.io
        import.io is a free web-based platform that puts the power of the machine readable web in your hands. Using our tools you can create an API or crawl an entire website in a fraction of the time of traditional methods, no coding required.
        BeautifulSoup
        It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work.
        Kimono
        You don't need to write any code or install any software to extract data with Kimono. The easiest way to use Kimono is to add our bookmarklet to your browser's bookmark bar. Then go to the website you want to get data from and click the bookmarklet. Select the data you want and Kimono does the rest. We take care of hosting the APIs that you build with Kimono and running them on the schedule you specify. Use the API output in JSON or as CSV files that you can easily paste into a spreadsheet.
        Portia
        Portia is an open source tool that lets you get data from websites. It facilitates and automates the process of data extraction. This visual web scraper works straight from your browser, so you don't need to download or install anything.
        Apifier
        Apifier is a hosted web crawler for developers that enables them to extract data from any website using a few simple lines of JavaScript and rapidly build new apps on top of existing third-party web apps and data sources.
        See all alternatives
        Decisions about Scrapy and Selenium
        No stack decisions found
        Interest over time
        Reviews of Scrapy and Selenium
        Review ofSeleniumSelenium

        we at www.cloudqa.io use selenium hub and node architecture to execute our test cases its a very powerful tool to automate web applications.

        How developers use Scrapy and Selenium
        Avatar of Kang Hyeon Ku
        Kang Hyeon Ku uses SeleniumSelenium

        가장 강력한 웹 UI / UX 테스트 툴 이지만, 문제도 많다. iframe 안에 있는 건 잘 select 가 안되기도 하고, 화면이 작아 컴포넌트가 화면에 보이지 않으면 또 select 할 수 없다. 하지만 브라우저를 있는 그대로 제어 할 수 있다는 게 가장 큰 매력이다.

        ps. selenide 와 같이 쓰면 코드가 많이 개선된다.

        Avatar of samdfonseca
        samdfonseca uses SeleniumSelenium

        Maintaining and expanding automated regression tests is one of my primary responsibilities at CrowdTwist. We use Selenium to test the front-end parts of our platform.

        Avatar of Ralic Lo
        Ralic Lo uses SeleniumSelenium

        Used Selenium web-driver for automated QA testing in continuous integration and continuous deployment.

        Avatar of Owen Miller
        Owen Miller uses SeleniumSelenium
        • Navigating our site for tests
        • Navigating other sites that discriminate against non-human visitors
        Avatar of Ataccama
        Ataccama uses SeleniumSelenium

        Selenium is a great way to automatically test if a build is sane and can be deployed.

        How much does Scrapy cost?
        How much does Selenium cost?
        Pricing unavailable
        Pricing unavailable
        News about Scrapy
        More news