Alternatives to Scrapy logo

Alternatives to Scrapy

Selenium, import.io, BeautifulSoup, Puppeteer, and ParseHub are the most popular alternatives and competitors to Scrapy.
172
168
+ 1
0

What is Scrapy and what are its top alternatives?

It is the most popular web scraping framework in Python. An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.
Scrapy is a tool in the Web Scraping API category of a tech stack.
Scrapy is an open source tool with 41.2K GitHub stars and 9.3K GitHub forks. Here’s a link to Scrapy's open source repository on GitHub

Top Alternatives to Scrapy

  • Selenium

    Selenium

    Selenium automates browsers. That's it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well. ...

  • import.io

    import.io

    import.io is a free web-based platform that puts the power of the machine readable web in your hands. Using our tools you can create an API or crawl an entire website in a fraction of the time of traditional methods, no coding required. ...

  • BeautifulSoup

    BeautifulSoup

    It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work. ...

  • Puppeteer

    Puppeteer

    Puppeteer is a Node library which provides a high-level API to control headless Chrome over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome. ...

  • ParseHub

    ParseHub

    Web Scraping and Data Extraction ParseHub is a free and powerful web scraping tool. With our advanced web scraper, extracting data is as easy as clicking on the data you need. ParseHub lets you turn any website into a spreadsheet or API w ...

  • Portia

    Portia

    Portia is an open source tool that lets you get data from websites. It facilitates and automates the process of data extraction. This visual web scraper works straight from your browser, so you don't need to download or install anything. ...

  • Kimono

    Kimono

    You don't need to write any code or install any software to extract data with Kimono. The easiest way to use Kimono is to add our bookmarklet to your browser's bookmark bar. Then go to the website you want to get data from and click the bookmarklet. Select the data you want and Kimono does the rest. We take care of hosting the APIs that you build with Kimono and running them on the schedule you specify. Use the API output in JSON or as CSV files that you can easily paste into a spreadsheet. ...

  • Apify

    Apify

    Apify is a platform that enables developers to create, customize and run cloud-based programs called actors that can, among other things, be used to extract data from any website using a few lines of JavaScript. ...

Scrapy alternatives & related posts

Selenium logo

Selenium

10.1K
7.6K
519
Web Browser Automation
10.1K
7.6K
+ 1
519
PROS OF SELENIUM
  • 169
    Automates browsers
  • 154
    Testing
  • 101
    Essential tool for running test automation
  • 24
    Record-Playback
  • 24
    Remote Control
  • 8
    Data crawling
  • 7
    Supports end to end testing
  • 6
    Functional testing
  • 6
    Easy set up
  • 4
    The Most flexible monitoring system
  • 3
    Easy to integrate with build tools
  • 3
    End to End Testing
  • 2
    Integration Tests
  • 2
    Comparing the performance selenium is faster than jasm
  • 2
    Record and playback
  • 2
    Compatible with Python
  • 2
    Easy to scale
  • 0
    Integrated into Selenium-Jupiter framework
CONS OF SELENIUM
  • 7
    Flaky tests
  • 2
    Slow as needs to make browser (even with no gui)

related Selenium posts

Kamil Kowalski
Lead Architect at Fresha · | 27 upvotes · 989.1K views

When you think about test automation, it’s crucial to make it everyone’s responsibility (not just QA Engineers'). We started with Selenium and Java, but with our platform revolving around Ruby, Elixir and JavaScript, QA Engineers were left alone to automate tests. Cypress was the answer, as we could switch to JS and simply involve more people from day one. There's a downside too, as it meant testing on Chrome only, but that was "good enough" for us + if really needed we can always cover some specific cases in a different way.

See more
Benjamin Poon
QA Manager - Engineering at HBC Digital · | 8 upvotes · 691.1K views

For our digital QA organization to support a complex hybrid monolith/microservice architecture, our team took on the lofty goal of building out a commonized UI test automation framework. One of the primary requisites included a technical minimalist threshold such that an engineer or analyst with fundamental knowledge of JavaScript could automate their tests with greater ease. Just to list a few: - Nightwatchjs - Selenium - Cucumber - GitHub - Go.CD - Docker - ExpressJS - React - PostgreSQL

With this structure, we're able to combine the automation efforts of each team member into a centralized repository while also providing new relevant metrics to business owners.

See more
import.io logo

import.io

34
73
23
Extract data from the web
34
73
+ 1
23
PROS OF IMPORT.IO
  • 7
    Easy setup
  • 5
    Free lead generation tool
  • 5
    Native desktop app
  • 3
    Continuous updates
  • 3
    Features based on users suggestions
CONS OF IMPORT.IO
    Be the first to leave a con

    related import.io posts

    BeautifulSoup logo

    BeautifulSoup

    55
    61
    3
    A Python library for pulling data out of HTML and XML files
    55
    61
    + 1
    3
    PROS OF BEAUTIFULSOUP
    • 2
      Parsed html even when poorly formed
    • 1
      It just works
    CONS OF BEAUTIFULSOUP
      Be the first to leave a con

      related BeautifulSoup posts

      Shared insights
      on
      ParseHub
      BeautifulSoup

      Which tool is best for webscrapping, BeautifulSoup or ParseHub???????????

      See more
      Puppeteer logo

      Puppeteer

      312
      426
      21
      Headless Chrome Node API
      312
      426
      + 1
      21
      PROS OF PUPPETEER
      • 8
        Scriptable web browser
      • 8
        Very well documented
      • 5
        Promise based
      CONS OF PUPPETEER
      • 7
        Chrome only

      related Puppeteer posts

      Raziel Alron
      Automation Engineer at Tipalti · | 7 upvotes · 91.1K views

      Currently, we are using Protractor in our project. Since Protractor isn't updated anymore, we are looking for a new tool. The strongest suggestions are WebdriverIO or Puppeteer. Please help me figure out what tool would make the transition fastest and easiest. Please note that Protractor uses its own locator system, and we want the switch to be as simple as possible. Thank you!

      See more

      I work in a company building web apps with AngularJS. I started using Selenium for tests automation, as I am more familiar with Python. However, I found some difficulties, like the impossibility of using IDs and fixed lists of classes, ending up with using xpaths most, which unfortunately could change with fixes and modifications in the code.

      So, I started using Puppeteer, but I am still learning. It seems easier to find elements on the webpage, even if the creation and managing of arrays of elements seem to be a little bit more complicated than in Selenium, but it could be also due to my poor knowledge of JavaScript.

      Any comments on this comparison and also on comparisons with similar tools are welcome! :)

      See more
      ParseHub logo

      ParseHub

      23
      55
      14
      Turn dynamic websites into APIs
      23
      55
      + 1
      14
      PROS OF PARSEHUB
      • 5
        Great support
      • 4
        Easy setup
      • 3
        Complex websites
      • 2
        Native Desktop App
      CONS OF PARSEHUB
        Be the first to leave a con

        related ParseHub posts

        Shared insights
        on
        ParseHub
        BeautifulSoup

        Which tool is best for webscrapping, BeautifulSoup or ParseHub???????????

        See more
        Portia logo

        Portia

        20
        54
        0
        Visual web scraping tool that lets you extract data without writing a single line of code
        20
        54
        + 1
        0
        PROS OF PORTIA
          Be the first to leave a pro
          CONS OF PORTIA
            Be the first to leave a con

            related Portia posts

            Kimono logo

            Kimono

            14
            28
            5
            Turn websites into structured APIs from your browser in seconds
            14
            28
            + 1
            5
            PROS OF KIMONO
            • 2
              Easy setup
            • 1
              Extracting data from ecommerce sites
            • 1
              Integrate API with web application
            • 1
              Data Scraping
            CONS OF KIMONO
              Be the first to leave a con

              related Kimono posts

              Apify logo

              Apify

              13
              53
              4
              Cloud-based web scraping tool for developers
              13
              53
              + 1
              4
              PROS OF APIFY
              • 4
                Perfect for Heavy Java Script Websites
              CONS OF APIFY
                Be the first to leave a con

                related Apify posts