Alternatives to Kubeflow logo

Alternatives to Kubeflow

TensorFlow, Apache Spark, MLflow, Airflow, and Polyaxon are the most popular alternatives and competitors to Kubeflow.
48
146
+ 1
5

What is Kubeflow and what are its top alternatives?

The Kubeflow project is dedicated to making Machine Learning on Kubernetes easy, portable and scalable by providing a straightforward way for spinning up best of breed OSS solutions.
Kubeflow is a tool in the Machine Learning Tools category of a tech stack.
Kubeflow is an open source tool with 8.7K GitHub stars and 1.4K GitHub forks. Here鈥檚 a link to Kubeflow's open source repository on GitHub

Kubeflow alternatives & related posts

related TensorFlow posts

Conor Myhrvold
Conor Myhrvold
Tech Brand Mgr, Office of CTO at Uber | 6 upvotes 637.1K views
atUber TechnologiesUber Technologies
TensorFlow
TensorFlow
Keras
Keras
PyTorch
PyTorch

Why we built an open source, distributed training framework for TensorFlow , Keras , and PyTorch:

At Uber, we apply deep learning across our business; from self-driving research to trip forecasting and fraud prevention, deep learning enables our engineers and data scientists to create better experiences for our users.

TensorFlow has become a preferred deep learning library at Uber for a variety of reasons. To start, the framework is one of the most widely used open source frameworks for deep learning, which makes it easy to onboard new users. It also combines high performance with an ability to tinker with low-level model details鈥攆or instance, we can use both high-level APIs, such as Keras, and implement our own custom operators using NVIDIA鈥檚 CUDA toolkit.

Uber has introduced Michelangelo (https://eng.uber.com/michelangelo/), an internal ML-as-a-service platform that democratizes machine learning and makes it easy to build and deploy these systems at scale. In this article, we pull back the curtain on Horovod, an open source component of Michelangelo鈥檚 deep learning toolkit which makes it easier to start鈥攁nd speed up鈥攄istributed deep learning projects with TensorFlow:

https://eng.uber.com/horovod/

(Direct GitHub repo: https://github.com/uber/horovod)

See more
StackShare Editors
StackShare Editors
Cassandra
Cassandra
Apache Spark
Apache Spark
TensorFlow
TensorFlow

In mid-2015, Uber began exploring ways to scale ML across the organization, avoiding ML anti-patterns while standardizing workflows and tools. This effort led to Michelangelo.

Michelangelo consists of a mix of open source systems and components built in-house. The primary open sourced components used are HDFS, Spark, Samza, Cassandra, MLLib, XGBoost, and TensorFlow.

!

See more

related Apache Spark posts

Eric Colson
Eric Colson
Chief Algorithms Officer at Stitch Fix | 19 upvotes 899.4K views
atStitch FixStitch Fix
Kafka
Kafka
PostgreSQL
PostgreSQL
Amazon S3
Amazon S3
Apache Spark
Apache Spark
Presto
Presto
Python
Python
R Language
R Language
PyTorch
PyTorch
Docker
Docker
Amazon EC2 Container Service
Amazon EC2 Container Service
#AWS
#Etl
#ML
#DataScience
#DataStack
#Data

The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.

Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).

At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.

For more info:

#DataScience #DataStack #Data

See more
Conor Myhrvold
Conor Myhrvold
Tech Brand Mgr, Office of CTO at Uber | 7 upvotes 457.6K views
atUber TechnologiesUber Technologies
Kafka
Kafka
Kafka Manager
Kafka Manager
Hadoop
Hadoop
Apache Spark
Apache Spark
GitHub
GitHub

Why we built Marmaray, an open source generic data ingestion and dispersal framework and library for Apache Hadoop :

Built and designed by our Hadoop Platform team, Marmaray is a plug-in-based framework built on top of the Hadoop ecosystem. Users can add support to ingest data from any source and disperse to any sink leveraging the use of Apache Spark . The name, Marmaray, comes from a tunnel in Turkey connecting Europe and Asia. Similarly, we envisioned Marmaray within Uber as a pipeline connecting data from any source to any sink depending on customer preference:

https://eng.uber.com/marmaray-hadoop-ingestion-open-source/

(Direct GitHub repo: https://github.com/uber/marmaray Kafka Kafka Manager )

See more
MLflow logo

MLflow

37
112
2
37
112
+ 1
2
An open source machine learning platform
MLflow logo
MLflow
VS
Kubeflow logo
Kubeflow
Airflow logo

Airflow

557
701
39
557
701
+ 1
39
A platform to programmaticaly author, schedule and monitor data pipelines, by Airbnb
Airflow logo
Airflow
VS
Kubeflow logo
Kubeflow
Polyaxon logo

Polyaxon

6
22
0
6
22
+ 1
0
An enterprise-grade open source platform for building, training, and monitoring large scale deep learning applications.
    Be the first to leave a pro
    Polyaxon logo
    Polyaxon
    VS
    Kubeflow logo
    Kubeflow
    Keras logo

    Keras

    534
    489
    11
    534
    489
    + 1
    11
    Deep Learning library for Theano and TensorFlow
    Keras logo
    Keras
    VS
    Kubeflow logo
    Kubeflow

    related Keras posts

    Conor Myhrvold
    Conor Myhrvold
    Tech Brand Mgr, Office of CTO at Uber | 6 upvotes 637.1K views
    atUber TechnologiesUber Technologies
    TensorFlow
    TensorFlow
    Keras
    Keras
    PyTorch
    PyTorch

    Why we built an open source, distributed training framework for TensorFlow , Keras , and PyTorch:

    At Uber, we apply deep learning across our business; from self-driving research to trip forecasting and fraud prevention, deep learning enables our engineers and data scientists to create better experiences for our users.

    TensorFlow has become a preferred deep learning library at Uber for a variety of reasons. To start, the framework is one of the most widely used open source frameworks for deep learning, which makes it easy to onboard new users. It also combines high performance with an ability to tinker with low-level model details鈥攆or instance, we can use both high-level APIs, such as Keras, and implement our own custom operators using NVIDIA鈥檚 CUDA toolkit.

    Uber has introduced Michelangelo (https://eng.uber.com/michelangelo/), an internal ML-as-a-service platform that democratizes machine learning and makes it easy to build and deploy these systems at scale. In this article, we pull back the curtain on Horovod, an open source component of Michelangelo鈥檚 deep learning toolkit which makes it easier to start鈥攁nd speed up鈥攄istributed deep learning projects with TensorFlow:

    https://eng.uber.com/horovod/

    (Direct GitHub repo: https://github.com/uber/horovod)

    See more
    scikit-learn logo

    scikit-learn

    529
    483
    25
    529
    483
    + 1
    25
    Easy-to-use and general-purpose machine learning in Python
    scikit-learn logo
    scikit-learn
    VS
    Kubeflow logo
    Kubeflow
    PyTorch logo

    PyTorch

    418
    442
    15
    418
    442
    + 1
    15
    A deep learning framework that puts Python first
    PyTorch logo
    PyTorch
    VS
    Kubeflow logo
    Kubeflow

    related PyTorch posts

    Conor Myhrvold
    Conor Myhrvold
    Tech Brand Mgr, Office of CTO at Uber | 6 upvotes 637.1K views
    atUber TechnologiesUber Technologies
    TensorFlow
    TensorFlow
    Keras
    Keras
    PyTorch
    PyTorch

    Why we built an open source, distributed training framework for TensorFlow , Keras , and PyTorch:

    At Uber, we apply deep learning across our business; from self-driving research to trip forecasting and fraud prevention, deep learning enables our engineers and data scientists to create better experiences for our users.

    TensorFlow has become a preferred deep learning library at Uber for a variety of reasons. To start, the framework is one of the most widely used open source frameworks for deep learning, which makes it easy to onboard new users. It also combines high performance with an ability to tinker with low-level model details鈥攆or instance, we can use both high-level APIs, such as Keras, and implement our own custom operators using NVIDIA鈥檚 CUDA toolkit.

    Uber has introduced Michelangelo (https://eng.uber.com/michelangelo/), an internal ML-as-a-service platform that democratizes machine learning and makes it easy to build and deploy these systems at scale. In this article, we pull back the curtain on Horovod, an open source component of Michelangelo鈥檚 deep learning toolkit which makes it easier to start鈥攁nd speed up鈥攄istributed deep learning projects with TensorFlow:

    https://eng.uber.com/horovod/

    (Direct GitHub repo: https://github.com/uber/horovod)

    See more