What is Scrapy and what are its top alternatives?
Top Alternatives to Scrapy
- Selenium
Selenium automates browsers. That's it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well. ...
- import.io
import.io is a free web-based platform that puts the power of the machine readable web in your hands. Using our tools you can create an API or crawl an entire website in a fraction of the time of traditional methods, no coding required. ...
- BeautifulSoup
It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work. ...
- Puppeteer
Puppeteer is a Node library which provides a high-level API to control headless Chrome over the DevTools Protocol. It can also be configured to use full (non-headless) Chrome. ...
- Apify
Apify is a platform that enables developers to create, customize and run cloud-based programs called actors that can, among other things, be used to extract data from any website using a few lines of JavaScript. ...
- ParseHub
Web Scraping and Data Extraction ParseHub is a free and powerful web scraping tool. With our advanced web scraper, extracting data is as easy as clicking on the data you need. ParseHub lets you turn any website into a spreadsheet or API w ...
- Octoparse
It is a free client-side Windows web scraping software that turns unstructured or semi-structured data from websites into structured data sets, no coding necessary. Extracted data can be exported as API, CSV, Excel or exported into a database. ...
- Portia
Portia is an open source tool that lets you get data from websites. It facilitates and automates the process of data extraction. This visual web scraper works straight from your browser, so you don't need to download or install anything. ...
Scrapy alternatives & related posts
- Automates browsers175
- Testing154
- Essential tool for running test automation101
- Record-Playback24
- Remote Control24
- Data crawling8
- Supports end to end testing7
- Easy set up6
- Functional testing6
- The Most flexible monitoring system4
- End to End Testing3
- Easy to integrate with build tools3
- Comparing the performance selenium is faster than jasm2
- Record and playback2
- Compatible with Python2
- Easy to scale2
- Integration Tests2
- Integrated into Selenium-Jupiter framework0
- Flaky tests8
- Slow as needs to make browser (even with no gui)4
- Update browser drivers2
related Selenium posts
When you think about test automation, it’s crucial to make it everyone’s responsibility (not just QA Engineers'). We started with Selenium and Java, but with our platform revolving around Ruby, Elixir and JavaScript, QA Engineers were left alone to automate tests. Cypress was the answer, as we could switch to JS and simply involve more people from day one. There's a downside too, as it meant testing on Chrome only, but that was "good enough" for us + if really needed we can always cover some specific cases in a different way.
For our digital QA organization to support a complex hybrid monolith/microservice architecture, our team took on the lofty goal of building out a commonized UI test automation framework. One of the primary requisites included a technical minimalist threshold such that an engineer or analyst with fundamental knowledge of JavaScript could automate their tests with greater ease. Just to list a few: - Nightwatchjs - Selenium - Cucumber - GitHub - Go.CD - Docker - ExpressJS - React - PostgreSQL
With this structure, we're able to combine the automation efforts of each team member into a centralized repository while also providing new relevant metrics to business owners.
- Easy setup8
- Native desktop app5
- Free lead generation tool5
- Continuous updates3
- Features based on users suggestions3
related import.io posts
- Parsed html even when poorly formed3
- It just works1
related BeautifulSoup posts
- Very well documented10
- Scriptable web browser10
- Promise based6
- Chrome only10
related Puppeteer posts
Currently, we are using Protractor in our project. Since Protractor isn't updated anymore, we are looking for a new tool. The strongest suggestions are WebdriverIO or Puppeteer. Please help me figure out what tool would make the transition fastest and easiest. Please note that Protractor uses its own locator system, and we want the switch to be as simple as possible. Thank you!
I work in a company building web apps with AngularJS. I started using Selenium for tests automation, as I am more familiar with Python. However, I found some difficulties, like the impossibility of using IDs and fixed lists of classes, ending up with using xpaths most, which unfortunately could change with fixes and modifications in the code.
So, I started using Puppeteer, but I am still learning. It seems easier to find elements on the webpage, even if the creation and managing of arrays of elements seem to be a little bit more complicated than in Selenium, but it could be also due to my poor knowledge of JavaScript.
Any comments on this comparison and also on comparisons with similar tools are welcome! :)
- Perfect for Heavy Java Script Websites4
related Apify posts
- Great support6
- Easy setup5
- Complex websites5
- Native Desktop App3
related ParseHub posts
Octoparse
- Cloud extraction3
- Easy to use3
- API2
- Great support1
- Web Scraping Template1
- Web Scraping Template1
- Auto-detection1
- Great support0