Why developers like Apache NiFi

What is Apache NiFi?

An easy to use, powerful, and reliable system to process and distribute data. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic.

Apache NiFi is a tool in the Message Queue category of a tech stack.

Who uses Apache NiFi?

Companies

42 companies reportedly use Apache NiFi in their tech stacks, including Hepsiburada, ViaVarejo, and QuintoAndar.

Hepsiburada

ViaVarejo

QuintoAndar

bigin

Wongnai

bol.com

EasyCrédito

Haptik

Startup Bakery

Developers

296 developers on StackShare have stated that they use Apache NiFi.

spurpose

My Stack

Stackspace

My Stack

Pack & Hold

Personal

My Stack

Professional Stack

Apache NiFi Integrations

MongoDB, Amazon S3, Kafka, Linux, and Amazon SQS are some of the popular tools that integrate with Apache NiFi. Here's a list of all 9 tools that integrate with Apache NiFi.

MongoDB

Amazon S3

Kafka

Linux

Amazon SQS

Amazon SNS

macOS

Apache Hive

Kylo

Pros of Apache NiFi

Visual Data Flows using Directed Acyclic Graphs (DAGs)

Free (Open Source)

Simple-to-use

Scalable horizontally as well as vertically

Reactive with back-pressure

Fast prototyping

Bi-directional channels

End-to-end security between all nodes

Built-in graphical user interface

Can handle messages up to gigabytes in size

Data provenance

Lots of documentation

Hbase support

Support for custom Processor in Java

Hive support

Kudu support

Slack integration

Lot of articles

Decisions about Apache NiFi

Here are some stack decisions, common use cases and reviews by companies and developers who chose Apache NiFi in their tech stack.

John Calandra

Data Manager at The Garrett Group · Dec 9, 2020 | 8 upvotes · 367.5K views

Needs advice

Linux

pgAdmin

and

PostgreSQL

There is a question coming... I am using Oracle VirtualBox to spawn 3 Ubuntu Linux virtual machines (VM). VM1 is being used as a data lake - just a place to store flat files. VM2 hosts Apache NiFi. VM3 hosts PostgreSQL. I have built a NiFi pipeline that reads flat files on VM1 and then pipes the data over to and inserts it into the Postgresql database. I left this setup alone for a while, and then something hiccupped on VM3, and I had to rebuild it. Now I cannot make a remote connection to Postgresql on VM3. I was using pgAdmin3 on VM3, but it kept throwing errors - I found out it went end-of-life in 2018 and uninstalled it. pgAdmin4 is out, but for some reason, I cannot get the APT utility to find/install it. I am trying to figure out the pgAdmin4 install problem and looking for a good alternative for pgAdmin4 that I can use to diagnose the remote database connection problem. Does anyone have any suggestions? Thanks in advance.

Sathya Ganapathy

Jun 11, 2020 | 4 upvotes · 658.4K views

Needs advice

Airflow

and

Apache NiFi

I am looking for the best tool to orchestrate #ETL workflows in non-Hadoop environments, mainly for regression testing use cases. Would Airflow or Apache NiFi be a good fit for this purpose?

For example, I want to run an Informatica ETL job and then run an SQL task as a dependency, followed by another task from Jira. What tool is best suited to set up such a pipeline?

See all decisions

Apache NiFi's Features

Web-based user interface
Highly configurable
Data Provenance
Designed for extension
Secure

Apache NiFi Alternatives & Comparisons

What are some alternatives to Apache NiFi?

Kafka

Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design.

Apache Storm

Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm has many use cases: realtime analytics, online machine learning, continuous computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked it at over a million tuples processed per second per node. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate.

Logstash

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for later use (like, for searching). If you store them in Elasticsearch, you can view and analyze them with Kibana.

Apache Camel

An open source Java framework that focuses on making integration easier and more accessible to developers.

Apache Spark

Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.

See all alternatives

Related Comparisons