Alternatives to Apache NiFi logo

Alternatives to Apache NiFi

Kafka, Apache Storm, Logstash, Apache Camel, and Apache Spark are the most popular alternatives and competitors to Apache NiFi.
28
13
+ 1
4

What is Apache NiFi and what are its top alternatives?

An easy to use, powerful, and reliable system to process and distribute data. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
Apache NiFi is a tool in the Stream Processing category of a tech stack.

Apache NiFi alternatives & related posts

related Kafka posts

Eric Colson
Eric Colson
Chief Algorithms Officer at Stitch Fix · | 19 upvotes · 209.1K views
atStitch FixStitch Fix
Amazon EC2 Container Service
Amazon EC2 Container Service
Docker
Docker
PyTorch
PyTorch
R
R
Python
Python
Presto
Presto
Apache Spark
Apache Spark
Amazon S3
Amazon S3
PostgreSQL
PostgreSQL
Kafka
Kafka
#Data
#DataStack
#DataScience
#ML
#Etl
#AWS

The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.

Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).

At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.

For more info:

#DataScience #DataStack #Data

See more
John Kodumal
John Kodumal
CTO at LaunchDarkly · | 15 upvotes · 98.8K views
atLaunchDarklyLaunchDarkly
Kafka
Kafka
Amazon Kinesis
Amazon Kinesis
Redis
Redis
Amazon EC2
Amazon EC2
Amazon ElastiCache
Amazon ElastiCache
Consul
Consul
Patroni
Patroni
TimescaleDB
TimescaleDB
PostgreSQL
PostgreSQL
Amazon RDS
Amazon RDS

As we've evolved or added additional infrastructure to our stack, we've biased towards managed services. Most new backing stores are Amazon RDS instances now. We do use self-managed PostgreSQL with TimescaleDB for time-series data—this is made HA with the use of Patroni and Consul.

We also use managed Amazon ElastiCache instances instead of spinning up Amazon EC2 instances to run Redis workloads, as well as shifting to Amazon Kinesis instead of Kafka.

See more
Apache Storm logo

Apache Storm

127
111
18
127
111
+ 1
18
Distributed and fault-tolerant realtime computation
Apache Storm logo
Apache Storm
VS
Apache NiFi logo
Apache NiFi

related Apache Storm posts

Marc Bollinger
Marc Bollinger
Infra & Data Eng Manager at Lumosity · | 4 upvotes · 38K views
atLumosityLumosity
Pulsar
Pulsar
Redis
Redis
Heron
Heron
Apache Storm
Apache Storm
Scala
Scala
Kafka
Kafka
Ruby
Ruby
Node.js
Node.js

Lumosity is home to the world's largest cognitive training database, a responsibility we take seriously. For most of the company's history, our analysis of user behavior and training data has been powered by an event stream--first a simple Node.js pub/sub app, then a heavyweight Ruby app with stronger durability. Both supported decent throughput and latency, but they lacked some major features supported by existing open-source alternatives: replaying existing messages (also lacking in most message queue-based solutions), scaling out many different readers for the same stream, the ability to leverage existing solutions for reading and writing, and possibly most importantly: the ability to hire someone externally who already had expertise.

We ultimately migrated to Kafka in early- to mid-2016, citing both industry trends in companies we'd talked to with similar durability and throughput needs, the extremely strong documentation and community. We pored over Kyle Kingsbury's Jepsen post (https://aphyr.com/posts/293-jepsen-Kafka), as well as Jay Kreps' follow-up (http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen), talked at length with Confluent folks and community members, and still wound up running parallel systems for quite a long time, but ultimately, we've been very, very happy. Understanding the internals and proper levers takes some commitment, but it's taken very little maintenance once configured. Since then, the Confluent Platform community has grown and grown; we've gone from doing most development using custom Scala consumers and producers to being 60/40 Kafka Streams/Connects.

We originally looked into Storm / Heron , and we'd moved on from Redis pub/sub. Heron looks great, but we already had a programming model across services that was more akin to consuming a message consumers than required a topology of bolts, etc. Heron also had just come out while we were starting to migrate things, and the community momentum and direction of Kafka felt more substantial than the older Storm. If we were to start the process over again today, we might check out Pulsar , although the ecosystem is much younger.

To find out more, read our 2017 engineering blog post about the migration!

See more

related Logstash posts

Tymoteusz Paul
Tymoteusz Paul
Devops guy at X20X Development LTD · | 12 upvotes · 195.6K views
Amazon EC2
Amazon EC2
LXC
LXC
CircleCI
CircleCI
Docker
Docker
Git
Git
Vault
Vault
Apache Maven
Apache Maven
Slack
Slack
Jenkins
Jenkins
TeamCity
TeamCity
Logstash
Logstash
Kibana
Kibana
Elasticsearch
Elasticsearch
Ansible
Ansible
VirtualBox
VirtualBox
Vagrant
Vagrant

Often enough I have to explain my way of going about setting up a CI/CD pipeline with multiple deployment platforms. Since I am a bit tired of yapping the same every single time, I've decided to write it up and share with the world this way, and send people to read it instead ;). I will explain it on "live-example" of how the Rome got built, basing that current methodology exists only of readme.md and wishes of good luck (as it usually is ;)).

It always starts with an app, whatever it may be and reading the readmes available while Vagrant and VirtualBox is installing and updating. Following that is the first hurdle to go over - convert all the instruction/scripts into Ansible playbook(s), and only stopping when doing a clear vagrant up or vagrant reload we will have a fully working environment. As our Vagrant environment is now functional, it's time to break it! This is the moment to look for how things can be done better (too rigid/too lose versioning? Sloppy environment setup?) and replace them with the right way to do stuff, one that won't bite us in the backside. This is the point, and the best opportunity, to upcycle the existing way of doing dev environment to produce a proper, production-grade product.

I should probably digress here for a moment and explain why. I firmly believe that the way you deploy production is the same way you should deploy develop, shy of few debugging-friendly setting. This way you avoid the discrepancy between how production work vs how development works, which almost always causes major pains in the back of the neck, and with use of proper tools should mean no more work for the developers. That's why we start with Vagrant as developer boxes should be as easy as vagrant up, but the meat of our product lies in Ansible which will do meat of the work and can be applied to almost anything: AWS, bare metal, docker, LXC, in open net, behind vpn - you name it.

We must also give proper consideration to monitoring and logging hoovering at this point. My generic answer here is to grab Elasticsearch, Kibana, and Logstash. While for different use cases there may be better solutions, this one is well battle-tested, performs reasonably and is very easy to scale both vertically (within some limits) and horizontally. Logstash rules are easy to write and are well supported in maintenance through Ansible, which as I've mentioned earlier, are at the very core of things, and creating triggers/reports and alerts based on Elastic and Kibana is generally a breeze, including some quite complex aggregations.

If we are happy with the state of the Ansible it's time to move on and put all those roles and playbooks to work. Namely, we need something to manage our CI/CD pipelines. For me, the choice is obvious: TeamCity. It's modern, robust and unlike most of the light-weight alternatives, it's transparent. What I mean by that is that it doesn't tell you how to do things, doesn't limit your ways to deploy, or test, or package for that matter. Instead, it provides a developer-friendly and rich playground for your pipelines. You can do most the same with Jenkins, but it has a quite dated look and feel to it, while also missing some key functionality that must be brought in via plugins (like quality REST API which comes built-in with TeamCity). It also comes with all the common-handy plugins like Slack or Apache Maven integration.

The exact flow between CI and CD varies too greatly from one application to another to describe, so I will outline a few rules that guide me in it: 1. Make build steps as small as possible. This way when something breaks, we know exactly where, without needing to dig and root around. 2. All security credentials besides development environment must be sources from individual Vault instances. Keys to those containers should exist only on the CI/CD box and accessible by a few people (the less the better). This is pretty self-explanatory, as anything besides dev may contain sensitive data and, at times, be public-facing. Because of that appropriate security must be present. TeamCity shines in this department with excellent secrets-management. 3. Every part of the build chain shall consume and produce artifacts. If it creates nothing, it likely shouldn't be its own build. This way if any issue shows up with any environment or version, all developer has to do it is grab appropriate artifacts to reproduce the issue locally. 4. Deployment builds should be directly tied to specific Git branches/tags. This enables much easier tracking of what caused an issue, including automated identifying and tagging the author (nothing like automated regression testing!).

Speaking of deployments, I generally try to keep it simple but also with a close eye on the wallet. Because of that, I am more than happy with AWS or another cloud provider, but also constantly peeking at the loads and do we get the value of what we are paying for. Often enough the pattern of use is not constantly erratic, but rather has a firm baseline which could be migrated away from the cloud and into bare metal boxes. That is another part where this approach strongly triumphs over the common Docker and CircleCI setup, where you are very much tied in to use cloud providers and getting out is expensive. Here to embrace bare-metal hosting all you need is a help of some container-based self-hosting software, my personal preference is with Proxmox and LXC. Following that all you must write are ansible scripts to manage hardware of Proxmox, similar way as you do for Amazon EC2 (ansible supports both greatly) and you are good to go. One does not exclude another, quite the opposite, as they can live in great synergy and cut your costs dramatically (the heavier your base load, the bigger the savings) while providing production-grade resiliency.

See more
Tanya Bragin
Tanya Bragin
Product Lead, Observability at Elastic · | 10 upvotes · 34.4K views
atElasticElastic
Kibana
Kibana
Logstash
Logstash
Elasticsearch
Elasticsearch

ELK Stack (Elasticsearch, Logstash, Kibana) is widely known as the de facto way to centralize logs from operational systems. The assumption is that Elasticsearch (a "search engine") is a good place to put text-based logs for the purposes of free-text search. And indeed, simply searching text-based logs for the word "error" or filtering logs based on a set of a well-known tags is extremely powerful, and is often where most users start.

See more
Apache Camel logo

Apache Camel

61
21
2
61
21
+ 1
2
A versatile open source integration framework
Apache Camel logo
Apache Camel
VS
Apache NiFi logo
Apache NiFi

related Apache Spark posts

Eric Colson
Eric Colson
Chief Algorithms Officer at Stitch Fix · | 19 upvotes · 209.1K views
atStitch FixStitch Fix
Amazon EC2 Container Service
Amazon EC2 Container Service
Docker
Docker
PyTorch
PyTorch
R
R
Python
Python
Presto
Presto
Apache Spark
Apache Spark
Amazon S3
Amazon S3
PostgreSQL
PostgreSQL
Kafka
Kafka
#Data
#DataStack
#DataScience
#ML
#Etl
#AWS

The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.

Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).

At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.

For more info:

#DataScience #DataStack #Data

See more
Conor Myhrvold
Conor Myhrvold
Tech Brand Mgr, Office of CTO at Uber · | 4 upvotes · 97.1K views
atUber TechnologiesUber Technologies
Kafka Manager
Kafka Manager
Kafka
Kafka
GitHub
GitHub
Apache Spark
Apache Spark
Hadoop
Hadoop

Why we built Marmaray, an open source generic data ingestion and dispersal framework and library for Apache Hadoop :

Built and designed by our Hadoop Platform team, Marmaray is a plug-in-based framework built on top of the Hadoop ecosystem. Users can add support to ingest data from any source and disperse to any sink leveraging the use of Apache Spark . The name, Marmaray, comes from a tunnel in Turkey connecting Europe and Asia. Similarly, we envisioned Marmaray within Uber as a pipeline connecting data from any source to any sink depending on customer preference:

https://eng.uber.com/marmaray-hadoop-ingestion-open-source/

(Direct GitHub repo: https://github.com/uber/marmaray Kafka Kafka Manager )

See more
Airflow logo

Airflow

308
240
16
308
240
+ 1
16
A platform to programmaticaly author, schedule and monitor data pipelines, by Airbnb
Airflow logo
Airflow
VS
Apache NiFi logo
Apache NiFi
Apache Beam logo

Apache Beam

19
11
0
19
11
+ 1
0
A unified programming model
    Be the first to leave a pro
    Apache Beam logo
    Apache Beam
    VS
    Apache NiFi logo
    Apache NiFi
    StreamSets logo

    StreamSets

    5
    2
    0
    5
    2
    + 1
    0
    Where DevOps Meets Data Integration
      Be the first to leave a pro
      StreamSets logo
      StreamSets
      VS
      Apache NiFi logo
      Apache NiFi

      related RabbitMQ posts

      James Cunningham
      James Cunningham
      Operations Engineer at Sentry · | 18 upvotes · 80.1K views
      atSentrySentry
      RabbitMQ
      RabbitMQ
      Celery
      Celery
      #MessageQueue

      As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

      Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

      #MessageQueue

      See more
      Tim Abbott
      Tim Abbott
      Founder at Zulip · | 9 upvotes · 43.9K views
      atZulipZulip
      Redis
      Redis
      Python
      Python
      RabbitMQ
      RabbitMQ

      We've been using RabbitMQ as Zulip's queuing system since we needed a queuing system. What I like about it is that it scales really well and has good libraries for a wide range of platforms, including our own Python. So aside from getting it running, we've had to put basically 0 effort into making it scale for our needs.

      However, there's several things that could be better about it: * It's error messages are absolutely terrible; if ever one of our users ends up getting an error with RabbitMQ (even for simple things like a misconfigured hostname), they always end up needing to get help from the Zulip team, because the errors logs are just inscrutable. As an open source project, we've handled this issue by really carefully scripting the installation to be a failure-proof configuration (in this case, setting the RabbitMQ hostname to 127.0.0.1, so that no user-controlled configuration can break it). But it was a real pain to get there and the process of determining we needed to do that caused a significant amount of pain to folks installing Zulip. * The pika library for Python takes a lot of time to startup a RabbitMQ connection; this means that Zulip server restarts are more disruptive than would be ideal. * It's annoying that you need to run the rabbitmqctl management commands as root.

      But overall, I like that it has clean, clear semanstics and high scalability, and haven't been tempted to do the work to migrate to something like Redis (which has its own downsides).

      See more

      related Amazon SQS posts

      Tim Specht
      Tim Specht
      ‎Co-Founder and CTO at Dubsmash · | 14 upvotes · 38.9K views
      atDubsmashDubsmash
      Google BigQuery
      Google BigQuery
      Amazon SQS
      Amazon SQS
      AWS Lambda
      AWS Lambda
      Amazon Kinesis
      Amazon Kinesis
      Google Analytics
      Google Analytics
      #BigDataAsAService
      #RealTimeDataProcessing
      #GeneralAnalytics
      #ServerlessTaskProcessing

      In order to accurately measure & track user behaviour on our platform we moved over quickly from the initial solution using Google Analytics to a custom-built one due to resource & pricing concerns we had.

      While this does sound complicated, it’s as easy as clients sending JSON blobs of events to Amazon Kinesis from where we use AWS Lambda & Amazon SQS to batch and process incoming events and then ingest them into Google BigQuery. Once events are stored in BigQuery (which usually only takes a second from the time the client sends the data until it’s available), we can use almost-standard-SQL to simply query for data while Google makes sure that, even with terabytes of data being scanned, query times stay in the range of seconds rather than hours. Before ingesting their data into the pipeline, our mobile clients are aggregating events internally and, once a certain threshold is reached or the app is going to the background, sending the events as a JSON blob into the stream.

      In the past we had workers running that continuously read from the stream and would validate and post-process the data and then enqueue them for other workers to write them to BigQuery. We went ahead and implemented the Lambda-based approach in such a way that Lambda functions would automatically be triggered for incoming records, pre-aggregate events, and write them back to SQS, from which we then read them, and persist the events to BigQuery. While this approach had a couple of bumps on the road, like re-triggering functions asynchronously to keep up with the stream and proper batch sizes, we finally managed to get it running in a reliable way and are very happy with this solution today.

      #ServerlessTaskProcessing #GeneralAnalytics #RealTimeDataProcessing #BigDataAsAService

      See more
      Praveen Mooli
      Praveen Mooli
      Technical Leader at Taylor and Francis · | 11 upvotes · 95.2K views
      MongoDB Atlas
      MongoDB Atlas
      Amazon S3
      Amazon S3
      Amazon DynamoDB
      Amazon DynamoDB
      Amazon RDS
      Amazon RDS
      Serverless
      Serverless
      Docker
      Docker
      Terraform
      Terraform
      Travis CI
      Travis CI
      GitHub
      GitHub
      RxJS
      RxJS
      Angular 2
      Angular 2
      AWS Lambda
      AWS Lambda
      Amazon SQS
      Amazon SQS
      Amazon SNS
      Amazon SNS
      Amazon Kinesis Firehose
      Amazon Kinesis Firehose
      Amazon Kinesis
      Amazon Kinesis
      Flask
      Flask
      Python
      Python
      ExpressJS
      ExpressJS
      Node.js
      Node.js
      Spring Boot
      Spring Boot
      Java
      Java
      #Data
      #Devops
      #Webapps
      #Eventsourcingframework
      #Microservices
      #Backend

      We are in the process of building a modern content platform to deliver our content through various channels. We decided to go with Microservices architecture as we wanted scale. Microservice architecture style is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many distributed servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single point of failure, which can help prevent large-scale outages. We also decided to use Event Driven Architecture pattern which is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.

      To build our #Backend capabilities we decided to use the following: 1. #Microservices - Java with Spring Boot , Node.js with ExpressJS and Python with Flask 2. #Eventsourcingframework - Amazon Kinesis , Amazon Kinesis Firehose , Amazon SNS , Amazon SQS, AWS Lambda 3. #Data - Amazon RDS , Amazon DynamoDB , Amazon S3 , MongoDB Atlas

      To build #Webapps we decided to use Angular 2 with RxJS

      #Devops - GitHub , Travis CI , Terraform , Docker , Serverless

      See more

      related Celery posts

      James Cunningham
      James Cunningham
      Operations Engineer at Sentry · | 18 upvotes · 80.1K views
      atSentrySentry
      RabbitMQ
      RabbitMQ
      Celery
      Celery
      #MessageQueue

      As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

      Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

      #MessageQueue

      See more
      ActiveMQ logo

      ActiveMQ

      188
      220
      51
      188
      220
      + 1
      51
      A message broker written in Java together with a full JMS client
      ActiveMQ logo
      ActiveMQ
      VS
      Apache NiFi logo
      Apache NiFi

      related ActiveMQ posts

      Naushad Warsi
      Naushad Warsi
      software developer at klingelnberg · | 1 upvotes · 38.1K views
      RabbitMQ
      RabbitMQ
      ActiveMQ
      ActiveMQ

      I use ActiveMQ because RabbitMQ have stopped giving the support for AMQP 1.0 or above version and the earlier version of AMQP doesn't give the functionality to support OAuth.

      If OAuth is not required and we can go with AMQP 0.9 then i still recommend rabbitMq.

      See more
      ZeroMQ logo

      ZeroMQ

      140
      124
      49
      140
      124
      + 1
      49
      Fast, lightweight messaging library that allows you to design complex communication system without much effort
      ZeroMQ logo
      ZeroMQ
      VS
      Apache NiFi logo
      Apache NiFi
      MQTT logo

      MQTT

      75
      35
      0
      75
      35
      + 1
      0
      A machine-to-machine Internet of Things connectivity protocol
        Be the first to leave a pro
        MQTT logo
        MQTT
        VS
        Apache NiFi logo
        Apache NiFi
        WCF logo

        WCF

        38
        16
        0
        38
        16
        + 1
        0
        A runtime and a set of APIs for building connected, service-oriented applications
          Be the first to leave a pro
          WCF logo
          WCF
          VS
          Apache NiFi logo
          Apache NiFi
          Kafka Manager logo

          Kafka Manager

          38
          45
          0
          38
          45
          + 1
          0
          A tool for managing Apache Kafka, developed by Yahoo
            Be the first to leave a pro
            Kafka Manager logo
            Kafka Manager
            VS
            Apache NiFi logo
            Apache NiFi

            related Kafka Manager posts

            Conor Myhrvold
            Conor Myhrvold
            Tech Brand Mgr, Office of CTO at Uber · | 4 upvotes · 97.1K views
            atUber TechnologiesUber Technologies
            Kafka Manager
            Kafka Manager
            Kafka
            Kafka
            GitHub
            GitHub
            Apache Spark
            Apache Spark
            Hadoop
            Hadoop

            Why we built Marmaray, an open source generic data ingestion and dispersal framework and library for Apache Hadoop :

            Built and designed by our Hadoop Platform team, Marmaray is a plug-in-based framework built on top of the Hadoop ecosystem. Users can add support to ingest data from any source and disperse to any sink leveraging the use of Apache Spark . The name, Marmaray, comes from a tunnel in Turkey connecting Europe and Asia. Similarly, we envisioned Marmaray within Uber as a pipeline connecting data from any source to any sink depending on customer preference:

            https://eng.uber.com/marmaray-hadoop-ingestion-open-source/

            (Direct GitHub repo: https://github.com/uber/marmaray Kafka Kafka Manager )

            See more