To handle its growing observability needs, Postmates created and open sourced Cernan, a telemetry and logging aggregation server. Ceran is built on Rust and Lua, and can ingest data from many sources and then push or exposes what it’s collected to many destinations, or “sinks.” It can also create or manipulate in-flight data with programmable Lua filters.

Sentry's event processing pipeline, which is responsible for handling all of the ingested event data that makes it through to our offline task processing, is written primarily in Python.

For particularly intense code paths, like our source map processing pipeline, we have begun re-writing those bits in Rust. Rust’s lack of garbage collection makes it a particularly convenient language for embedding in Python. It allows us to easily build a Python extension where all memory is managed from the Python side (if the Python wrapper gets collected by the Python GC we clean up the Rust object as well).

Stitch is run entirely on AWS. All of our transactional databases are run with Amazon RDS, and we rely on Amazon S3 for data persistence in various stages of our pipeline. Our product integrates with Amazon Redshift as a data destination, and we also use Redshift as an internal data warehouse (powered by Stitch, of course).

The majority of our services run on stateless Amazon EC2 instances that are managed by AWS OpsWorks. We recently introduced Kubernetes into our infrastructure to run the scheduled jobs that execute Singer code to extract data from various sources. Although we tend to be wary of shiny new toys, Kubernetes has proven to be a good fit for this problem, and its stability, strong community and helpful tooling have made it easy for us to incorporate into our operations.

While we continue to be happy with Clojure for our internal services, we felt that its relatively narrow adoption could impede Singer's growth. We chose Python both because it is well suited to the task, and it seems to have reached critical mass among data engineers. All that being said, the Singer spec is language agnostic, and integrations and libraries have been developed in JavaScript, Go, and Clojure.

One important decision for delivering a platform independent solution with low memory footprint and minimal dependencies was the choice of the programming language. We considered a few from Python (there was already a reasonably large Python code base at Thumbtack), to Go (we were taking our first steps with it), and even Rust (too immature at the time).

We ended up writing it in C. It was easy to meet all requirements with only one external dependency for implementing the web server, clearly no challenges running it on any of the Linux distributions we were maintaining, and arguably the implementation with the smallest memory footprint given the choices above.

Initially, I wrote my text adventure game in C++, but I later rewrote my project in Rust. It was an incredibly easier process to use Rust to create a faster, more robust, and bug-free project.

One difficulty with the C++ language is the lack of safety, helpful error messages, and useful abstractions when compared to languages like Rust. Rust would display a helpful error message at compile time, while C++ would often display "Segmentation fault (core dumped)" or wall of STL errors in the middle. While I would frequently push buggy code to my C++ repository, Rust enabled me to only even submit fully functional code.

Along with the actual language, Rust also included useful tools such as rustup and cargo to aid in building projects, IDE tooling, managing dependencies, and cross-compiling. This was a refreshing alternative to the difficult CMake and tools of the same nature.

I adopted Clojure and ClojureScript because:

• it's 1 language, multiple platforms.
• Simple syntax.
• Designed to avoid unwanted side effects and bugs.
• Immutable data-structures.
• Compact code, very expressive.
• Source code is data.
• It has super-flexible macro.
• Has metadata.
• Interoperability with JavaScript, Java and C#.

At FlowStack we write most of our backend in Go. Go is a well thought out language, with all the right compromises for speedy development of speedy and robust software. It's tooling is part of what makes Go such a great language. Testing and benchmarking is built into the language, in a way that makes it easy to ensure correctness and high performance. In most cases you can get more performance out of Rust and C or C++, but getting everything right is more cumbersome.