How Uber developed the open source, end-to-end distributed tracing Jaeger , now a CNCF project:

Distributed tracing is quickly becoming a must-have component in the tools that organizations use to monitor their complex, microservice-based architectures. At Uber, our open source distributed tracing system Jaeger saw large-scale internal adoption throughout 2016, integrated into hundreds of microservices and now recording thousands of traces every second.

Here is the story of how we got here, from investigating off-the-shelf solutions like Zipkin, to why we switched from pull to push architecture, and how distributed tracing will continue to evolve:

https://eng.uber.com/distributed-tracing/

(GitHub Pages : https://www.jaegertracing.io/, GitHub: https://github.com/jaegertracing/jaeger)

Bindings/Operator: Python Java Node.js Go C++ Kubernetes JavaScript OpenShift C# Apache Spark

At our company, and I've noticed a lot of other ones... application developers and dev-ops people tend to use Ruby and our statisticians and data scientists love Python . Like most companies, our stack is kind of split that way. Ruby is used as glue in most of our production systems ( Java being the main backend language), and then all of our data scientists and their various pipelines tend towards Python

Initially, I wrote my text adventure game in C++, but I later rewrote my project in Rust. It was an incredibly easier process to use Rust to create a faster, more robust, and bug-free project.

One difficulty with the C++ language is the lack of safety, helpful error messages, and useful abstractions when compared to languages like Rust. Rust would display a helpful error message at compile time, while C++ would often display "Segmentation fault (core dumped)" or wall of STL errors in the middle. While I would frequently push buggy code to my C++ repository, Rust enabled me to only even submit fully functional code.

Along with the actual language, Rust also included useful tools such as rustup and cargo to aid in building projects, IDE tooling, managing dependencies, and cross-compiling. This was a refreshing alternative to the difficult CMake and tools of the same nature.

Multiple systems means there is a requirement to cart data across them.

Started off with Talend scripts. This was great as what we initially had were PHP/Python script - allowed for a more systematic approach to ETL.

But ended up with a massive repository of scripts, complex crontab entries and regular failures due to memory issues.

Using Stitch or similar services is a better approach: - no need to worry about the infrastructure needed for the ETL processes - a more formal mapping of data from source to destination as opposed to script developer doing his/her voodoo magic - lot of common sources and destination integrations are already builtin and out of the box

etl @{etlasaservice}|topic:1323|

I use Visual Studio Code because at this time is a mature software and I can do practically everything using it.

• It's free and open source: The project is hosted on GitHub and it’s free to download, fork, modify and contribute to the project.

• Multi-platform: You can download binaries for different platforms, included Windows (x64), MacOS and Linux (.rpm and .deb packages)

• LightWeight: It runs smoothly in different devices. It has an average memory and CPU usage. Starts almost immediately and it’s very stable.

• Extended language support: Supports by default the majority of the most used languages and syntax like JavaScript, HTML, C#, Swift, Java, PHP, Python and others. Also, VS Code supports different file types associated to projects like .ini, .properties, XML and JSON files.

• Integrated tools: Includes an integrated terminal, debugger, problem list and console output inspector. The project navigator sidebar is simple and powerful: you can manage your files and folders with ease. The command palette helps you find commands by text. The search widget has a powerful auto-complete feature to search and find your files.

• Extensible and configurable: There are many extensions available for every language supported, including syntax highlighters, IntelliSense and code completion, and debuggers. There are also extension to manage application configuration and architecture like Docker and Jenkins.

• Integrated with Git: You can visually manage your project repositories, pull, commit and push your changes, and easy conflict resolution.( there is support for SVN (Subversion) users by plugin)

Multiple systems means there is a requirement to cart data across them.

Started off with Talend scripts. This was great as what we initially had were PHP/Python script - allowed for a more systematic approach to ETL.

But ended up with a massive repository of scripts, complex crontab entries and regular failures due to memory issues.

Using Stitch or similar services is a better approach: - no need to worry about the infrastructure needed for the ETL processes - a more formal mapping of data from source to destination as opposed to script developer doing his/her voodoo magic - lot of common sources and destination integrations are already builtin and out of the box

The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.

Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).

At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.

For more info:

• Our Algorithms Tour: https://algorithms-tour.stitchfix.com/
• Our blog: https://multithreaded.stitchfix.com/blog/
• Careers: https://multithreaded.stitchfix.com/careers/

#DataScience #DataStack #Data

Django or NodeJS? Hi, I’m thinking about which software I should use for my web-app. What about Node.js or Django for the back-end? I want to create an online preparation course for the final school exams in my country. At the beginning for maths. The course should contain tutorials and a lot of exercises of different types. E.g. multiple choice, user text/number input and drawing tasks. The exercises should change (different levels) with the learning progress. Wrong questions should asked again with different numbers. I also want a score system and statistics. So far, I have got only limited web development skills. (some HTML, CSS, Bootstrap and Wordpress). I don’t know JavaScript or Python.

Possible pros for Python / Django: - easy syntax, easier to learn for me as a beginner - fast development, earlier release - libraries for mathematical and scientific computation

Possible pros for JavaScript / Node.js: - great performance, better choice for real time applications: user should get the answer for a question quickly

Which software would you use in my case? Are my arguments for Python/NodeJS right? Which kind of database would you use?

Thank you for your answer!

Node.js JavaScript Django Python

Go is a high performance language with simple syntax / semantics. Although it is not as expressive as some other languages, it's still a great language for backend development.

Python is expressive and battery-included, and pre-installed in most linux distros, making it a great language for scripting.

PostgreSQL: Rock-solid RDBMS with NoSQL support.

TypeScript saves you from all nonsense semantics of JavaScript , LOL.

NATS: fast message queue and easy to deploy / maintain.

Docker makes deployment painless.

Git essential tool for collaboration and source management.

I'm #Fullstack here and work with Vue.js, React and Node.js in some projects but also C# for other clients. Also started learning Python. And all this with just one tool!: #Vscode I have used Atom and Sublime Text in the past and they are very good too, but for me now is just vscode. I think the combination of vscode with the free available extensions that the community is creating makes a powerful tool and that's why vscode became the most popular IDE for software development. You can match it to your own needs in a couple of minutes. Did I mention you can style it your way? Amazing tool!

Our most used programming languages are JavaScript / Node.js for it's lightweight and fast use, PHP because everyone knows it, HTML5 because you can't live without it and Sass to write great CSS. Occasionally, we use nginx as a web server and proxy, React for our UX, PostgreSQL as fast relational database, Ubuntu as server OS, ES6 and TypeScript for Node, Google Compute Engine for our infrastructure, and Socket.IO and Electron for specific use cases. We also use Python for some of our backends.

We are in the process of building a modern content platform to deliver our content through various channels. We decided to go with Microservices architecture as we wanted scale. Microservice architecture style is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many distributed servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single point of failure, which can help prevent large-scale outages. We also decided to use Event Driven Architecture pattern which is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.

To build our #Backend capabilities we decided to use the following: 1. #Microservices - Java with Spring Boot , Node.js with ExpressJS and Python with Flask 2. #Eventsourcingframework - Amazon Kinesis , Amazon Kinesis Firehose , Amazon SNS , Amazon SQS, AWS Lambda 3. #Data - Amazon RDS , Amazon DynamoDB , Amazon S3 , MongoDB Atlas

To build #Webapps we decided to use Angular 2 with RxJS

#Devops - GitHub , Travis CI , Terraform , Docker , Serverless

I love Python and JavaScript . You can do the same JavaScript async operations in Python by using asyncio. This is particularly useful when you need to do socket programming in Python. With streaming sockets, data can be sent or received at any time. In case your Python program is in the middle of executing some code, other threads can handle the new socket data. Libraries like asyncio implement multiple threads, so your Python program can work in an asynchronous fashion. PubNub makes bi-directional data streaming between devices even easier.

Python is a excellent tool for #DataScience , but up to now is very poor in #uxdesign . To do some design I'm using JavaScript and #nodejs , #electron stack. The possibility of use CSS 3 to draw interfaces is very awesome and fast. Unfortunatelly Python don't have (yet) a good way to make a #UXdesign .

At FlowStack we write most of our backend in Go. Go is a well thought out language, with all the right compromises for speedy development of speedy and robust software. It's tooling is part of what makes Go such a great language. Testing and benchmarking is built into the language, in a way that makes it easy to ensure correctness and high performance. In most cases you can get more performance out of Rust and C or C++, but getting everything right is more cumbersome.