One important decision for delivering a platform independent solution with low memory footprint and minimal dependencies was the choice of the programming language. We considered a few from Python (there was already a reasonably large Python code base at Thumbtack), to Go (we were taking our first steps with it), and even Rust (too immature at the time).

We ended up writing it in C. It was easy to meet all requirements with only one external dependency for implementing the web server, clearly no challenges running it on any of the Linux distributions we were maintaining, and arguably the implementation with the smallest memory footprint given the choices above.

As the WeWork footprint continued to expand, in mid-2018 the team began to explore the next generation of identity management to handle the global scale of the business.

The team decided to vet three languages for building microservices: Go, Kotlin, and Ruby. They compared the three by building a component of an identity system in each, and assessing the performance apples-to-apples.

After building out the systems and load testing each one, the team decided to implement the new system in Go for a few reasons. In addition to better performance under heavy loads, Go, according to the team, is a simpler language that will constrain developers to simpler code. Additionally, the development lifecycle is simpler with Go, since “there is little difference between running a service directly on a dev machine, to running it in a container, to running clustered instances of the service.”

In the implementation, they the Go grpc framework to handle various common infrastructure patterns, resulting in “in a clean common server pattern that we can reuse across our microservices.”

Ruby NLP C++ Grammar #BNF

At FriendlyData we had a Ruby-based pipeline for natural language processing. Our technology is centered around grammar-based natural language parsing, as well as various product features, and, as the core stack of the company historically is Ruby, the initial version of the pipeline was implemented in Ruby as well.

As we were entering the exponential growth phase, both technology- and product-wise, we looked into how could we speed up and extend the performance and flexibility of our [meta-]BNF-based parsing engine. Gradually, we built the pieces of the engine in C++.

Ultimately, the natural language parsing stack spans three universes and three software engineering paradigms: the declarative one, the functional one, and the imperative one. The imperative one was and remains implemented in Ruby, the functional one is implemented in a functional language (this part is under the NDA, while everything I am talking about here is part of the public talks we gave throughout 2017 and 2018), and the declarative part, which can loosely be thought of as being BNF-based, is now served by the C++ engine.

The C++ engine for the BNF part removed the immediate blockers, gave us 500x+ performance speedup, and enabled us to launch new product features, most notably query completions, suggestions, and spelling corrections.

I chose Sqreen because it provides an out-of-the-box Security as a Service solution to protect my customer data. I get full visibility over my application security in real-time and I reduce my risk against the most common threats. My customers are happy and I don't need to spend any engineering resources or time on this. We're only alerted when our attention is required and the data that is provided helps engineering teams easily remediate vulnerabilities. The platform grows with us and will allow us to have all the right tools in place when our first security engineer joins the company. Advanced security protections against business logic threats can then be implemented.

Installation was super easy on my Node.js and Ruby apps. But Sqreen also supports Python , Java , PHP and soon Go .

It integrates well with the tools I'm using every day Slack , PagerDuty and more.

Lumosity is home to the world's largest cognitive training database, a responsibility we take seriously. For most of the company's history, our analysis of user behavior and training data has been powered by an event stream--first a simple Node.js pub/sub app, then a heavyweight Ruby app with stronger durability. Both supported decent throughput and latency, but they lacked some major features supported by existing open-source alternatives: replaying existing messages (also lacking in most message queue-based solutions), scaling out many different readers for the same stream, the ability to leverage existing solutions for reading and writing, and possibly most importantly: the ability to hire someone externally who already had expertise.

We ultimately migrated to Kafka in early- to mid-2016, citing both industry trends in companies we'd talked to with similar durability and throughput needs, the extremely strong documentation and community. We pored over Kyle Kingsbury's Jepsen post (https://aphyr.com/posts/293-jepsen-Kafka), as well as Jay Kreps' follow-up (http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen), talked at length with Confluent folks and community members, and still wound up running parallel systems for quite a long time, but ultimately, we've been very, very happy. Understanding the internals and proper levers takes some commitment, but it's taken very little maintenance once configured. Since then, the Confluent Platform community has grown and grown; we've gone from doing most development using custom Scala consumers and producers to being 60/40 Kafka Streams/Connects.

We originally looked into Storm / Heron , and we'd moved on from Redis pub/sub. Heron looks great, but we already had a programming model across services that was more akin to consuming a message consumers than required a topology of bolts, etc. Heron also had just come out while we were starting to migrate things, and the community momentum and direction of Kafka felt more substantial than the older Storm. If we were to start the process over again today, we might check out Pulsar , although the ecosystem is much younger.

To find out more, read our 2017 engineering blog post about the migration!