Alternatives to Scraper API logo

Alternatives to Scraper API

Amazon S3, Google Cloud Storage, Amazon EBS, Azure Storage, and Minio are the most popular alternatives and competitors to Scraper API.
1
6
+ 1
0

What is Scraper API and what are its top alternatives?

It handles proxies, browsers, and CAPTCHAs for you, so you can scrape any web page with a simple API call. Get started with 1000 free API calls per month.
Scraper API is a tool in the Web Scraping API category of a tech stack.

Top Alternatives to Scraper API

Scraper API alternatives & related posts

related Amazon S3 posts

Ashish Singh
Ashish Singh
Tech Lead, Big Data Platform at Pinterest | 31 upvotes 381.8K views

To provide employees with the critical need of interactive querying, we鈥檝e worked with Presto, an open-source distributed SQL query engine, over the years. Operating Presto at Pinterest鈥檚 scale has involved resolving quite a few challenges like, supporting deeply nested and huge thrift schemas, slow/ bad worker detection and remediation, auto-scaling cluster, graceful cluster shutdown and impersonation support for ldap authenticator.

Our infrastructure is built on top of Amazon EC2 and we leverage Amazon S3 for storing our data. This separates compute and storage layers, and allows multiple compute clusters to share the S3 data.

We have hundreds of petabytes of data and tens of thousands of Apache Hive tables. Our Presto clusters are comprised of a fleet of 450 r4.8xl EC2 instances. Presto clusters together have over 100 TBs of memory and 14K vcpu cores. Within Pinterest, we have close to more than 1,000 monthly active users (out of total 1,600+ Pinterest employees) using Presto, who run about 400K queries on these clusters per month.

Each query submitted to Presto cluster is logged to a Kafka topic via Singer. Singer is a logging agent built at Pinterest and we talked about it in a previous post. Each query is logged when it is submitted and when it finishes. When a Presto cluster crashes, we will have query submitted events without corresponding query finished events. These events enable us to capture the effect of cluster crashes over time.

Each Presto cluster at Pinterest has workers on a mix of dedicated AWS EC2 instances and Kubernetes pods. Kubernetes platform provides us with the capability to add and remove workers from a Presto cluster very quickly. The best-case latency on bringing up a new worker on Kubernetes is less than a minute. However, when the Kubernetes cluster itself is out of resources and needs to scale up, it can take up to ten minutes. Some other advantages of deploying on Kubernetes platform is that our Presto deployment becomes agnostic of cloud vendor, instance types, OS, etc.

#BigData #AWS #DataScience #DataEngineering

See more
Simon Reymann
Simon Reymann
Senior Fullstack Developer at QUANTUSflow Software GmbH | 25 upvotes 724K views

Our whole DevOps stack consists of the following tools:

  • GitHub (incl. GitHub Pages/Markdown for Documentation, GettingStarted and HowTo's) for collaborative review and code management tool
  • Respectively Git as revision control system
  • SourceTree as Git GUI
  • Visual Studio Code as IDE
  • CircleCI for continuous integration (automatize development process)
  • Prettier / TSLint / ESLint as code linter
  • SonarQube as quality gate
  • Docker as container management (incl. Docker Compose for multi-container application management)
  • VirtualBox for operating system simulation tests
  • Kubernetes as cluster management for docker containers
  • Heroku for deploying in test environments
  • nginx as web server (preferably used as facade server in production environment)
  • SSLMate (using OpenSSL) for certificate management
  • Amazon EC2 (incl. Amazon S3) for deploying in stage (production-like) and production environments
  • PostgreSQL as preferred database system
  • Redis as preferred in-memory database/store (great for caching)

The main reason we have chosen Kubernetes over Docker Swarm is related to the following artifacts:

  • Key features: Easy and flexible installation, Clear dashboard, Great scaling operations, Monitoring is an integral part, Great load balancing concepts, Monitors the condition and ensures compensation in the event of failure.
  • Applications: An application can be deployed using a combination of pods, deployments, and services (or micro-services).
  • Functionality: Kubernetes as a complex installation and setup process, but it not as limited as Docker Swarm.
  • Monitoring: It supports multiple versions of logging and monitoring when the services are deployed within the cluster (Elasticsearch/Kibana (ELK), Heapster/Grafana, Sysdig cloud integration).
  • Scalability: All-in-one framework for distributed systems.
  • Other Benefits: Kubernetes is backed by the Cloud Native Computing Foundation (CNCF), huge community among container orchestration tools, it is an open source and modular tool that works with any OS.
See more
Google Cloud Storage logo

Google Cloud Storage

989
798
71
Durable and highly available object storage service
989
798
+ 1
71
PROS OF GOOGLE CLOUD STORAGE
CONS OF GOOGLE CLOUD STORAGE
    No cons available

    related Google Cloud Storage posts

    Aliadoc Team
    Aliadoc Team
    at aliadoc.com | 5 upvotes 465.4K views

    In #Aliadoc, we're exploring the crowdfunding option to get traction before launch. We are building a SaaS platform for website design customization.

    For the Admin UI and website editor we use React and we're currently transitioning from a Create React App setup to a custom one because our needs have become more specific. We use CloudFlare as much as possible, it's a great service.

    For routing dynamic resources and proxy tasks to feed websites to the editor we leverage CloudFlare Workers for improved responsiveness. We use Firebase for our hosting needs and user authentication while also using several Cloud Functions for Firebase to interact with other services along with Google App Engine and Google Cloud Storage, but also the Real Time Database is on the radar for collaborative website editing.

    We generally hate configuration but honestly because of the stage of our project we lack resources for doing heavy sysops work. So we are basically just relying on Serverless technologies as much as we can to do all server side processing.

    Visual Studio Code definitively makes programming a much easier and enjoyable task, we just love it. We combine it with Bitbucket for our source code control needs.

    See more
    Amazon EBS logo

    Amazon EBS

    623
    449
    81
    Block level storage volumes for use with Amazon EC2 instances.
    623
    449
    + 1
    81

    related Amazon EBS posts

    Medeti Vamsi Krishna
    Medeti Vamsi Krishna

    We are looking for a centralised monitoring solution for our application deployed on Amazon EKS. We would like to monitor using metrics from Kubernetes, AWS services (NeptuneDB, AWS Elastic Load Balancing (ELB), Amazon EBS, Amazon S3, etc) and application microservice's custom metrics.

    We are expected to use around 80 microservices (not replicas). I think a total of 200-250 microservices will be there in the system with 10-12 slave nodes.

    We tried Prometheus but it looks like maintenance is a big issue. We need to manage scaling, maintaining the storage, and dealing with multiple exporters and Grafana. I felt this itself needs few dedicated resources (at least 2-3 people) to manage. Not sure if I am thinking in the correct direction. Please confirm.

    You mentioned Datadog and Sysdig charges per host. Does it charge per slave node?

    See more
    Azure Storage logo

    Azure Storage

    451
    474
    50
    Reliable, economical cloud storage for data big and small
    451
    474
    + 1
    50
    Minio logo

    Minio

    182
    265
    18
    AWS S3 open source alternative written in Go
    182
    265
    + 1
    18
    Scrapy logo

    Scrapy

    113
    89
    0
    A fast high-level web crawling & scraping framework for Python
    113
    89
    + 1
    0
    PROS OF SCRAPY
      No pros available
      CONS OF SCRAPY
        No cons available
        Rackspace Cloud Files logo

        Rackspace Cloud Files

        59
        59
        25
        Store it on Cloud Files, serve it fast on Akamai's CDN
        59
        59
        + 1
        25
        iCloud logo

        iCloud

        44
        42
        0
        A cloud storage and cloud computing service
        44
        42
        + 1
        0
        PROS OF ICLOUD
          No pros available
          CONS OF ICLOUD
            No cons available

            related iCloud posts

            I'm looking for a tool or set of tools to enable searching across all of our platforms including Confluence and Jira, Zoho CRM, Gmail, Gdrive for business, Dropbox and iCloud.

            Any ideas. Something like X1? IBM Watson Discovery?

            (And local Disk of course)

            See more