Alternatives to Metaflow

Data science and engineering teams at Lyft maintain several big data pipelines that serve as the foundation for various types of analysis throughout the business.

Apache Airflow sits at the center of this big data infrastructure, allowing users to “programmatically author, schedule, and monitor data pipelines.” Airflow is an open source tool, and “Lyft is the very first Airflow adopter in production since the project was open sourced around three years ago.”

There are several key components of the architecture. A web UI allows users to view the status of their queries, along with an audit trail of any modifications the query. A metadata database stores things like job status and task instance status. A multi-process scheduler handles job requests, and triggers the executor to execute those tasks.

Airflow supports several executors, though Lyft uses CeleryExecutor to scale task execution in production. Airflow is deployed to three Amazon Auto Scaling Groups, with each associated with a celery queue.

Audit logs supplied to the web UI are powered by the existing Airflow audit logs as well as Flask signal.

Datadog, Statsd, Grafana, and PagerDuty are all used to monitor the Airflow system.

We are a young start-up with 2 developers and a team in India looking to choose our next ETL tool. We have a few processes in Azure Data Factory but are looking to switch to a better platform. We were debating Trifacta and Airflow. Or even staying with Azure Data Factory. The use case will be to feed data to front-end APIs.

Can you please advise which one to choose FastText Or Gensim, in terms of:

1. Operability with ML Ops tools such as MLflow, Kubeflow, etc.
2. Performance
3. Customization of Intermediate steps
4. FastText and Gensim both have the same underlying libraries
5. Use cases each one tries to solve
6. Unsupervised Vs Supervised dimensions
7. Ease of Use.

Please mention any other points that I may have missed here.

We are trying to standardise DevOps across both ML (model selection and deployment) and regular software. Want to minimise the number of tools we have to learn. Also want a scalable solution which is easy enough to start small - eg. on a powerful laptop and eventually be deployed at scale. MLflow vs Kubernetes (Kubeflow)?

Google Analytics is a great tool to analyze your traffic. To debug our software and ask questions, we love to use Postman and Stack Overflow. Google Drive helps our team to share documents. We're able to build our great products through the APIs by Google Maps, CloudFlare, Stripe, PayPal, Twilio, Let's Encrypt, and TensorFlow.

Why we built an open source, distributed training framework for TensorFlow , Keras , and PyTorch:

At Uber, we apply deep learning across our business; from self-driving research to trip forecasting and fraud prevention, deep learning enables our engineers and data scientists to create better experiences for our users.

TensorFlow has become a preferred deep learning library at Uber for a variety of reasons. To start, the framework is one of the most widely used open source frameworks for deep learning, which makes it easy to onboard new users. It also combines high performance with an ability to tinker with low-level model details—for instance, we can use both high-level APIs, such as Keras, and implement our own custom operators using NVIDIA’s CUDA toolkit.

Uber has introduced Michelangelo (https://eng.uber.com/michelangelo/), an internal ML-as-a-service platform that democratizes machine learning and makes it easy to build and deploy these systems at scale. In this article, we pull back the curtain on Horovod, an open source component of Michelangelo’s deep learning toolkit which makes it easier to start—and speed up—distributed deep learning projects with TensorFlow:

https://eng.uber.com/horovod/

(Direct GitHub repo: https://github.com/uber/horovod)

I already use DVC to keep track and store my datasets in my machine learning pipeline. I have also started to use MLflow to keep track of my experiments. However, I still don't know whether to use DVC for my model files or I use the MLflow artifact store for this purpose. Or maybe these two serve different purposes, and it may be good to do both! Can anyone help, please?

Can you please advise which one to choose FastText Or Gensim, in terms of:

1. Operability with ML Ops tools such as MLflow, Kubeflow, etc.
2. Performance
3. Customization of Intermediate steps
4. FastText and Gensim both have the same underlying libraries
5. Use cases each one tries to solve
6. Unsupervised Vs Supervised dimensions
7. Ease of Use.

Please mention any other points that I may have missed here.

The client-side stack of Shopify Admin has been a long journey. It started with HTML templates, jQuery and Prototype. We moved to Batman.js, our in-house Single-Page-Application framework (SPA), in 2013. Then, we re-evaluated our approach and moved back to statically rendered HTML and vanilla JavaScript. As the front-end ecosystem matured, we felt that it was time to rethink our approach again. Last year, we started working on moving Shopify Admin to React and TypeScript.

Many things have changed since the days of jQuery and Batman. JavaScript execution is much faster. We can easily render our apps on the server to do less work on the client, and the resources and tooling for developers are substantially better with React than we ever had with Batman.

#FrameworksFullStack #Languages

I'm planning to create a web application and also a mobile application to provide a very good shopping experience to the end customers. Shortly, my application will be aggregate the product details from difference sources and giving a clear picture to the user that when and where to buy that product with best in Quality and cost.

I have planned to develop this in many milestones for adding N number of features and I have picked my first part to complete the core part (aggregate the product details from different sources).

As per my work experience and knowledge, I have chosen the followings stacks to this mission.

UI: I would like to develop this application using React, React Router and React Native since I'm a little bit familiar on this and also most importantly these will help on developing both web and mobile apps. In addition, I'm gonna use the stacks JavaScript, jQuery, jQuery UI, jQuery Mobile, Bootstrap wherever required.

Service: I have planned to use Java as the main business layer language as I have 7+ years of experience on this I believe I can do better work using Java than other languages. In addition, I'm thinking to use the stacks Node.js.

Database and ORM: I'm gonna pick MySQL as DB and Hibernate as ORM since I have a piece of good knowledge and also work experience on this combination.

Search Engine: I need to deal with a large amount of product data and it's in-detailed info to provide enough details to end user at the same time I need to focus on the performance area too. so I have decided to use Solr as a search engine for product search and suggestions. In addition, I'm thinking to replace Solr by Elasticsearch once explored/reviewed enough about Elasticsearch.

Host: As of now, my plan to complete the application with decent features first and deploy it in a free hosting environment like Docker and Heroku and then once it is stable then I have planned to use the AWS products Amazon S3, EC2, Amazon RDS and Amazon Route 53. I'm not sure about Microsoft Azure that what is the specialty in it than Heroku and Amazon EC2 Container Service. Anyhow, I will do explore these once again and pick the best suite one for my requirement once I reached this level.

Build and Repositories: I have decided to choose Apache Maven and Git as these are my favorites and also so popular on respectively build and repositories.

Additional Utilities :) - I would like to choose Codacy for code review as their Startup plan will be very helpful to this application. I'm already experienced with Google CheckStyle and SonarQube even I'm looking something on Codacy.

Happy Coding! Suggestions are welcome! :)

Thanks, Ganesa

I was building a personal project that I needed to store items in a real time database. I am more comfortable with my Frontend skills than my backend so I didn't want to spend time building out anything in Ruby or Go.

I stumbled on Firebase by #Google, and it was really all I needed. It had realtime data, an area for storing file uploads and best of all for the amount of data I needed it was free!

I built out my application using tools I was familiar with, React for the framework, Redux.js to manage my state across components, and styled-components for the styling.

Now as this was a project I was just working on in my free time for fun I didn't really want to pay for hosting. I did some research and I found Netlify. I had actually seen them at #ReactRally the year before and deployed a Gatsby site to Netlify already.

Netlify was very easy to setup and link to my GitHub account you select a repo and pretty much with very little configuration you have a live site that will deploy every time you push to master.

With the selection of these tools I was able to build out my application, connect it to a realtime database, and deploy to a live environment all with $0 spent.

If you're looking to build out a small app I suggest giving these tools a go as you can get your idea out into the real world for absolutely no cost.

Your tech stack is solid for building a real-time messaging project.

React and React Native are excellent choices for the frontend, especially if you want to have both web and mobile versions of your application share code.

ExpressJS is an unopinionated framework that affords you the flexibility to use it's features at your term, which is a good start. However, I would recommend you explore Sails.js as well. Sails.js is built on top of Express.js and it provides additional features out of the box, especially the Websocket integration that your project requires.

Don't forget to set up Graphql codegen, this would improve your dev experience (Add Typescript, if you can too).

I don't know much about databases but you might want to consider using NO-SQL. I used Firebase real-time db and aws dynamo db on a few of my personal projects and I love they're easy to work with and offer more flexibility for a chat application.

Our whole Node.js backend stack consists of the following tools:

• Lerna as a tool for multi package and multi repository management
• npm as package manager
• NestJS as Node.js framework
• TypeScript as programming language
• ExpressJS as web server
• Swagger UI for visualizing and interacting with the API’s resources
• Postman as a tool for API development
• TypeORM as object relational mapping layer
• JSON Web Token for access token management

The main reason we have chosen Node.js over PHP is related to the following artifacts:

• Made for the web and widely in use: Node.js is a software platform for developing server-side network services. Well-known projects that rely on Node.js include the blogging software Ghost, the project management tool Trello and the operating system WebOS. Node.js requires the JavaScript runtime environment V8, which was specially developed by Google for the popular Chrome browser. This guarantees a very resource-saving architecture, which qualifies Node.js especially for the operation of a web server. Ryan Dahl, the developer of Node.js, released the first stable version on May 27, 2009. He developed Node.js out of dissatisfaction with the possibilities that JavaScript offered at the time. The basic functionality of Node.js has been mapped with JavaScript since the first version, which can be expanded with a large number of different modules. The current package managers (npm or Yarn) for Node.js know more than 1,000,000 of these modules.
• Fast server-side solutions: Node.js adopts the JavaScript "event-loop" to create non-blocking I/O applications that conveniently serve simultaneous events. With the standard available asynchronous processing within JavaScript/TypeScript, highly scalable, server-side solutions can be realized. The efficient use of the CPU and the RAM is maximized and more simultaneous requests can be processed than with conventional multi-thread servers.
• A language along the entire stack: Widely used frameworks such as React or AngularJS or Vue.js, which we prefer, are written in JavaScript/TypeScript. If Node.js is now used on the server side, you can use all the advantages of a uniform script language throughout the entire application development. The same language in the back- and frontend simplifies the maintenance of the application and also the coordination within the development team.
• Flexibility: Node.js sets very few strict dependencies, rules and guidelines and thus grants a high degree of flexibility in application development. There are no strict conventions so that the appropriate architecture, design structures, modules and features can be freely selected for the development.

Our whole Vue.js frontend stack (incl. SSR) consists of the following tools:

• Nuxt.js consisting of Vue CLI, Vue Router, vuex, Webpack and Sass (Bundler for HTML5, CSS 3), Babel (Transpiler for JavaScript),
• Vue Styleguidist as our style guide and pool of developed Vue.js components
• Vuetify as Material Component Framework (for fast app development)
• TypeScript as programming language
• Apollo / GraphQL (incl. GraphiQL) for data access layer (https://apollo.vuejs.org/)
• ESLint, TSLint and Prettier for coding style and code analyzes
• Jest as testing framework
• Google Fonts and Font Awesome for typography and icon toolkit
• NativeScript-Vue for mobile development

The main reason we have chosen Vue.js over React and AngularJS is related to the following artifacts:

• Empowered HTML. Vue.js has many similar approaches with Angular. This helps to optimize HTML blocks handling with the use of different components.
• Detailed documentation. Vue.js has very good documentation which can fasten learning curve for developers.
• Adaptability. It provides a rapid switching period from other frameworks. It has similarities with Angular and React in terms of design and architecture.
• Awesome integration. Vue.js can be used for both building single-page applications and more difficult web interfaces of apps. Smaller interactive parts can be easily integrated into the existing infrastructure with no negative effect on the entire system.
• Large scaling. Vue.js can help to develop pretty large reusable templates.
• Tiny size. Vue.js weights around 20KB keeping its speed and flexibility. It allows reaching much better performance in comparison to other frameworks.