Why developers like Hadoop

What is Hadoop?

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

Hadoop is a tool in the Databases category of a tech stack.

Hadoop is an open source tool with 12.6K GitHub stars and 7.8K GitHub forks. Here’s a link to Hadoop's open source repository on GitHub

Who uses Hadoop?

Companies

358 companies reportedly use Hadoop in their tech stacks, including Uber, Airbnb, and Shopify.

Uber

Airbnb

Shopify

Netflix

Spotify

Twitter

Slack

Delivery Hero

Developers

1630 developers on StackShare have stated that they use Hadoop.

Marin Software

freakynit

Oracle Cloud PaaS

Omneo

Game Big Data

AllTech

Fun-Box

Krishna

Munye

Hadoop Integrations

Datadog, Azure Cosmos DB, Presto, Apache Flink, and Oracle PL/SQL are some of the popular tools that integrate with Hadoop. Here's a list of all 41 tools that integrate with Hadoop.

Datadog

Azure Cosmos DB

Presto

Apache Flink

Oracle PL/SQL

Couchbase

Databricks

Apache Hive

Apache Zeppelin

Pros of Hadoop

Great ecosystem

One stack to rule them all

Great load balancer

Amazon aws

Java syntax

Decisions about Hadoop

Here are some stack decisions, common use cases and reviews by companies and developers who chose Hadoop in their tech stack.

Sathya Ganapathy

Jun 11, 2020 | 4 upvotes · 324K views

Shared insights

I am looking for the best tool to orchestrate #ETL workflows in non-Hadoop environments, mainly for regression testing use cases. Would Airflow or Apache NiFi be a good fit for this purpose?

For example, I want to run an Informatica ETL job and then run an SQL task as a dependency, followed by another task from Jira. What tool is best suited to set up such a pipeline?

Mr Tech

SVP CTO · Apr 22, 2021 | 4 upvotes · 64.6K views

Shared insights

Snowflake

Hadoop

MarkLogic

For a property and casualty insurance company, we currently use MarkLogic and Hadoop for our raw data lake. Trying to figure out how snowflake fits in the picture. Does anybody have some good suggestions/best practices for when to use and what data to store in Mark logic versus Snowflake versus a hadoop or all three of these platforms redundant with one another?

See all decisions

Blog Posts

Improving Efficiency and Reducing Runtime Using S3 Read Optimi...

Sep 1 2021 at 5:34PM

1163

Pinterest Visual Signals Infrastructure: Evolution from Lambda...

Nov 24 2020 at 7:01PM

1902

Powering Inclusive Search & Recommendations with Our New Visua...

Aug 26 2020 at 4:42PM

683

Powering Pinterest Ads Analytics with Apache Druid

Apr 8 2020 at 5:37PM

1720

Cultivating your Data Lake

Aug 28 2019 at 3:10AM

Segment

+16

2247

Scaling Wix to 60M Users - From Monolith to Microservices

May 29 2015 at 9:25AM

Wix

+35

17781

Jobs that mention Hadoop as a desired skillset

Software Engineer, Business Intelligence

Dublin, IE

View Job Details

University Grad Machine Learning Engineer - MASTERS ONLY (USA)

San Francisco, CA, US; Palo Alto, CA, US

View Job Details

Software Engineer, Stream Processing Platform

Palo Alto, CA, US

View Job Details

+11

Software Engineer, Signal Delivery

Toronto, ON, CA

View Job Details

Software Engineer, Data Warehouse

Dublin, IE

View Job Details

Software Engineer, Big Data

San Francisco, CA, US

View Job Details

See all jobs

Hadoop Alternatives & Comparisons

What are some alternatives to Hadoop?

Cassandra

Partitioning means that Cassandra can distribute your data across multiple machines in an application-transparent matter. Cassandra will automatically repartition as machines are added and removed from the cluster. Row store means that like relational databases, Cassandra organizes data by rows and columns. The Cassandra Query Language (CQL) is a close relative of SQL.

MongoDB

MongoDB stores data in JSON-like documents that can vary in structure, offering a dynamic, flexible schema. MongoDB was also designed for high availability and scalability, with built-in replication and auto-sharding.

Elasticsearch

Elasticsearch is a distributed, RESTful search and analytics engine capable of storing data and searching it in near real time. Elasticsearch, Kibana, Beats and Logstash are the Elastic Stack (sometimes called the ELK Stack).

Splunk

It provides the leading platform for Operational Intelligence. Customers use it to search, monitor, analyze and visualize machine data.

Snowflake

Snowflake eliminates the administration and management demands of traditional data warehouses and big data platforms. Snowflake is a true data warehouse as a service running on Amazon Web Services (AWS)—no infrastructure to manage and no knobs to turn.

See all alternatives

Related Comparisons