Our team is made up of both remote and local employees. After doing daily standups over Zoom for a while, we started looking for a better solution. Having a fixed time for video meetings was problematic due to time zone differences. And the information that was exchanged was lost to whoever missed the meeting, and not super accessible even if the video was recorded. We also had a goal of minimizing the number of video meetings in our workflow since they're a bit disruptive.
After some discussion, we landed on Jell. It works for a remote/hybrid team, since you can post your status any time. It integrates with Slack, so all posts are collected in one channel. This allows you to browse posts and access information much more easily than you would in video form. It also makes it easy to keep track of daily tasks via a todo-style checklist. We've been using Jell for several months now and it's worked out great for our team.
Standups #MeetingTools RemoteTeam
Lumosity is home to the world's largest cognitive training database, a responsibility we take seriously. For most of the company's history, our analysis of user behavior and training data has been powered by an event stream--first a simple Node.js pub/sub app, then a heavyweight Ruby app with stronger durability. Both supported decent throughput and latency, but they lacked some major features supported by existing open-source alternatives: replaying existing messages (also lacking in most message queue-based solutions), scaling out many different readers for the same stream, the ability to leverage existing solutions for reading and writing, and possibly most importantly: the ability to hire someone externally who already had expertise.
We ultimately migrated to Kafka in early- to mid-2016, citing both industry trends in companies we'd talked to with similar durability and throughput needs, the extremely strong documentation and community. We pored over Kyle Kingsbury's Jepsen post (https://aphyr.com/posts/293-jepsen-Kafka), as well as Jay Kreps' follow-up (http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen), talked at length with Confluent folks and community members, and still wound up running parallel systems for quite a long time, but ultimately, we've been very, very happy. Understanding the internals and proper levers takes some commitment, but it's taken very little maintenance once configured. Since then, the Confluent Platform community has grown and grown; we've gone from doing most development using custom Scala consumers and producers to being 60/40 Kafka Streams/Connects.
We originally looked into Storm / Heron , and we'd moved on from Redis pub/sub. Heron looks great, but we already had a programming model across services that was more akin to consuming a message consumers than required a topology of bolts, etc. Heron also had just come out while we were starting to migrate things, and the community momentum and direction of Kafka felt more substantial than the older Storm. If we were to start the process over again today, we might check out Pulsar , although the ecosystem is much younger.
To find out more, read our 2017 engineering blog post about the migration!
Our front-end team decided to use React Storybook for our primary React development environment. It allows us to write components in isolation without the need to fire up our Rails stack. When writing components in isolation; you can focus on styling, behaviour and prop design. It forces you to think about how your component is going to be used by others. React Storybook uses webpack and hot module reloading under the hood. This allows us to write components very quickly since it hot reloads in the browser as you code!
The knobs add-on is great for testing different edge cases for the component props. There is even an add-on that will auto-render and snapshot your components with every prop permutation allows by your defined knobs. These snapshots can then be part of your CI testing.
We have a step in our build process that publishes a static React Storybook site on our production server. This allows our entire team to interactively test components before they are integrated into larger features. Once we are happy with our components in isolation, we integrate them into connected feature components which are wired up to Apollo and GraphQL to provide the data and state.
There are heaps of React Storybook add-ons to checkout. If you aren't using it, you should be.
Why we spent several years building an open source, large-scale metrics alerting system, M3, built for Prometheus:
By late 2014, all services, infrastructure, and servers at Uber emitted metrics to a Graphite stack that stored them using the Whisper file format in a sharded Carbon cluster. We used Grafana for dashboarding and Nagios for alerting, issuing Graphite threshold checks via source-controlled scripts. While this worked for a while, expanding the Carbon cluster required a manual resharding process and, due to lack of replication, any single node’s disk failure caused permanent loss of its associated metrics. In short, this solution was not able to meet our needs as the company continued to grow.
To ensure the scalability of Uber’s metrics backend, we decided to build out a system that provided fault tolerant metrics ingestion, storage, and querying as a managed platform...
(GitHub : https://github.com/m3db/m3)
Looker , Stitch , Amazon Redshift , dbt
We recently moved our Data Analytics and Business Intelligence tooling to Looker . It's already helping us create a solid process for reusable SQL-based data modeling, with consistent definitions across the entire organizations. Looker allows us to collaboratively build these version-controlled models and push the limits of what we've traditionally been able to accomplish with analytics with a lean team.
For Data Engineering, we're in the process of moving from maintaining our own ETL pipelines on AWS to a managed ELT system on Stitch. We're also evaluating the command line tool, dbt to manage data transformations. Our hope is that Stitch + dbt will streamline the ELT bit, allowing us to focus our energies on analyzing data, rather than managing it.