FLO HEALTH, INC.

We train and deploy various ML algorithms to personalize the user experience in every part of the Flo app. While our first models were trained and served in a custom way, it quickly became hard to manage all the complex datasets and entities we deal with.

Therefore we’ve adopted the Tecton Feature Store. Tecton is one of the most advanced Feature Stores on the market. It allows to quickly explore and experiment with new features and datasets offline, while making it easy to serve the exact same data in real-time in a high-load environment.

The key benefits of using the Tecton Feature Store:

• A clear and unified feature definition framework - less place for bugs and inconsistency between training on historical data and real-time serving

• The feature definition code is covered with tests, the feature availability and freshness is monitored

• Online data serving is easy to run and manage - every feature service becomes an API endpoint that is launched with a single command

• Time-travel - we’re able to obtain the exact historical values of every feature we store and serve - this makes it easy to debug models retrospectively

• Features are reusable in multiple problems - we can collaborate across different domains, while keeping the same standards for ML engineering for the whole company

Tecton runs on our own AWS infrastructure, using Spark and DynamoDB as offline and online storages accordingly. Tecton was our first step on our way to a scalable, reliable and efficient ML infrastructure at Flo.

Fast data growth and the importance of data-driven decisions make the data catalog one of the core components of data management. Flo Health adheres to high standards in engineering, including data solutions. The company itself has grown rapidly in recent years, and the accompanying increase in data made it obvious that we needed a solution to issues of data ownership, quality, and discoverability, as well as data governance.

So how did we resolve these issues?

Background and data-related issues:

Data ownership and responsibilities

Most data pipelines and datasets were owned by product teams and data analysts, but some pipelines with complex calculation logic were owned by data engineers. There were also cases that had multiple owners, or none. On top of that, responsibilities for data owners were not clearly defined. This meant that some data pipelines were implemented using best data engineering standards, covered with high-quality tests, and routinely validated, but because other data owners did not consider those things to be their responsibility, that wasn’t happening across the board.

Data discoverability and observability

With growing volumes of data, complex data pipelines, changing data sources, and an increasing number of people dealing with data, data discovery and observability became a challenge. Cases when it was difficult to determine the business context of data, find proper data for analysis because similar data was stored in multiple tables, or understand downstream processes began to appear.

Data trustability and governance

We didn’t have a simple entry point that would let data users both find proper data for analysis and know whether this data is trustable or not (i.e., if it had been tested, by what type of tests, and when it had last been successfully tested). There also wasn’t a centralized place to find all de-identified PII data in the storage or an automatic mechanism to identify potential data noncompliance in terms of privacy ahead of time.

What we needed

So, we needed a single entry point for working with the data that could resolve all our data issues. Of course, we also had high technical expectations:

• Ability to integrate with various data sources: Glue, Databricks, Looker, etc.
• Rich data lineage
• REST API

In addition, we wanted the tool to have a clear and simple UI and UX so that it wouldn’t create constraints in data governance process adoption. Everyone in the company, regardless of technical skillset, needed to be able to easily gather insights from data.

There’s a multitude of solutions on the market:

• Open-source solutions: Amundsen, DataHub, Magda, Atlas, etc.
• Proprietary solutions: Alation, Atlan, Collibra, etc.
• Mono-cloud solutions: Google Cloud Data Catalog, Azure Data Catalog
• Data observability platforms: Datafold, Monte Carlo

However, not all of them could meet our needs and fit a reasonable budget. You can find an overview of high-level tools here: github.com/Alexkuva/awesome-data-catalogs. To make the decision, our data engineers performed a comprehensive analysis of available open-source and proprietary solutions and agreed to go with Atlan.

What we can do with Atlan

1. Collect and centralize all the company’s metadata in one place and add necessary technical and business information for each entity (e.g., table, column, dashboard).
2. Get transparency in data pipelines with the help of automated data lineage across multiple sources.
3. Achieve clarity on data ownership. All core tables contain the name of the responsible team owner in Atlan. People are also assigned as owners and experts.
4. Develop a business glossary and connect data with appropriate glossary terms to help users understand the context of data assets.
5. Integrate our data quality tests with Atlan metadata via REST API to automatically set the status of test executions to a particular table.
6. Run data profiling functionality to collect base statistical information about the data.
7. Query all tables directly from Atlan, save and share SQL queries, and collaborate on issues via integrated chat.
8. Comply with security and confidentiality regulations for data user management via access policies, user groups, and roles.
9. Auto-detect PII using provided column names and auto-glossary recommendation options.

This isn’t everything Atlan can do — just the functionalities that we’re currently using the most at Flo right now. It should be mentioned that we’ve been using Atlan for a little less than a year, and we’re currently in the process of data catalog adoption among users. So far, we haven’t faced any bottlenecks in the Atlan functionalities related to our needs. I’m excited to see how it goes.

As of 2024 we still believe that we made the right choice back in 2015 of Android SDK as the basis for our Android application.

Nowadays, we could choose from plenty of cross-platform SDK options, which would’ve probably saved us resources at the beginning of the product’s development life cycle. However, engineering resource utilization isn’t the only consideration for making decisions. If you wanted to create the best women’s health solution on the market, you would need to care about performance and seamless integration with operating system features too. The modern cross-platform SDKs have just begun to get closer to the native development option in that regard. The Kotlin Multiplatform Project is a good example of such a framework. Unfortunately, because it hasn't been around for a long time, it still has plenty of issues, so it currently doesn't fit our needs. However, we might consider it in the future. All in all, I believe that we made the right choice.

Over time, Android engineering best practices, tools, and the operating system itself evolved, giving developers multiple ways to implement the same features more effectively, both in terms of engineering team performance and device resource utilization. Our team evolved as well: We’ve come a long way from a single Android developer to a dozen feature teams that need to work on the same codebase simultaneously without stepping on each other's toes. We began caring more about cycle time because one can’t successfully compete by delivering value slowly.

For our dev team, these changes prompted a request to update the codebase in order to deliver value faster and increase the speed of new Android features adoption, raising the overall level of quality at the same time.

We began with the modularization of our Android application. Using the power of the Gradle build system, we split our application into 70+ shared core modules and 30+ independent feature modules. Such a huge step required the revision of the application’s architecture. One could say that we moved to clean architecture; however, I would say that we use architecture driven by common software engineering principles like SOLID, DRY, KISS, etc. On the presentation layer, we switched from the MVP to the MVVM pattern. Implementation of this pattern, powered by the Jetpack Lifecycle components, simplifies Android component lifecycle management and increases the reusability of the code.

Supporting such a setup would be barely possible without a dependency injection (DI) implementation. We settled on Dagger 2. This DI framework supports compile-time graph validation, multibinding, and scoping support. Apart from that, it offers two ways to wire up individual components into a single graph: subcomponents and component dependencies, each good for its purpose. At Flo, we prefer component dependencies, as they better isolate the features and positively impact the build speed, but we use subcomponents closer to the leaves of the dependency graph as well.

Though we still have Java code in the project, Kotlin has become our main programming language. Compared to Java, it has multiple advantages:

• Improved type system, which, for example, makes it possible to avoid the “billion-dollar mistake” in the majority of cases
• Rich and mature standard library, which provides solutions for many typical tasks out of the box and minimizes the need for extra utilities
• Advanced features to better fit the open-closed principle (for example, extension functions and removal of checked exceptions let us improve the extendability of solutions)
• The syntax sugar, which simply lets you write code faster (it’s hard to imagine modern Android development without data classes, sealed classes, delegates, etc.) We attempt to use Kotlin wherever possible. Our build scripts are written in it, and we also migrate the good old bash scripts onto KScript.

Another huge step in Kotlin adoption is the migration from RxJava to the Kotlin coroutines. RxJava is a superb framework for event-based and asynchronous programming. However, it is not the best choice for asynchronous programming. In that regard, Kotlin coroutines seem like a much wiser choice, offering more effective resource utilization, more readable error stack traces, finer control over the execution scope and the syntax, which looks almost identical to the synchronous code. In its main area of usage — event-based programming — RxJava has also begun to lose ground. Being written in Java, it does not support Kotlin’s type system well. Besides, many of its operators are synchronous by design, which can limit developers. Driven by the Kotlin coroutines, Flow addresses both of these drawbacks and we found it perfectly fits our needs.

Perhaps the most noticeable sign that the above changes were not taken in vain is that you can now use Flo on your smartwatch powered by Android Wear. This is the second Flo app for the Android platform, and it effectively reuses the codebase of the mobile app. One of the core advantages of the Flo Watch app lies in Wear Health Services. It allows us to effectively and securely collect health-related data from the user’s device, if a user so chooses, and utilize it to improve the precision of cycle estimation.

But we won't stop chasing innovation!

Even though we migrated to ViewBinding, enjoying the extra type safety and reduced amount of the boilerplate code, we couldn’t pass by the Jetpack Compose framework. It allows us to use Kotlin power to construct UI, reduces code duplication, increases reusability of the UI components, and unblocks building complex view hierarchies with less performance penalty.

Finally, what about recent Android features support? Well, we continuously improve the app in that sense. Like most teams, we rely on different Jetpack, Firebase, and Play Services libraries to achieve that goal. We use them to move work to the background, implement push notifications, integrate billing, and many other little things, all of which improve the overall UX or let us reach out to users more effectively. However, we also invest in first-party tooling. In an effort to ensure secure and transparent management of user data, we implemented our own solutions for A/B testing, remote configuration management, logging, and analytics.

What about quality? Developers are responsible for the quality of created solutions. To ensure that we use modern tools and approaches:

• We chose Detekt and Android Lint for static code analysis. These frameworks prevent many issues from coming up in production by analyzing the codebase during compile time. They are capable of finding the most common problems in Kotlin and Android-related code, ensuring the whole team follows the same code style. When those frameworks do not provide the necessary checks out of the box, we implement them by ourselves.
• The above two frameworks are used both locally and in the continuous integration pipelines. However, in the latter, we additionally utilize the Sonarcloud tool. It provides extended complexity, security, and potential bug checks, which are run in the cloud.
• To ensure that the code meets the requirements, we use multiple layers of automated testing. Our test pyramid includes unit tests, which use the JUnit5 platform, and E2E tests powered by Espresso framework. Together, these two approaches to testing allow developers to get feedback fast while at the same time ensuring that features work as expected end-to-end.

Mid-2019, we started a search for a better solution for all of our analytical needs. Our existing analytical infrastructure had started to fall apart; it was incapable of keeping up with the rapid growth of the data we were collecting and analyzing at the time. Trino (previously known as Presto SQL) became our prime candidate since it had a feature set we needed; was actively maintained by and active community (and also commercial companies like Facebook and Starburst); had seamless integration with our BI tools; and last but not least, was successfully used by the biggest companies in tech — Facebook, Uber, Netflix, AirBnB, and many others. It remains at the heart of analytics at Flo, helping us to make truly data-driven decisions ever since.

​One particular aspect of Trino that distinguishes it from the competition is its ability to integrate a large variety of data sources thanks to its Connector API. With this feature, you can do things like expose data from Kafka topics as Trino tables or query your favorite SQL/NoSQL database. Trino's performance is also great because it allows us to process workloads requiring relatively low latency and process long-running queries effectively.

​A bit of trivia: Since all of our infrastructure is deployed in AWS, at the time, it was tempting for us to deploy Presto as a part of Amazon EMR; however, we went with a different approach. There were two flavors of Presto at the time (and still are): the original Facebook version and a fork named PrestoSQL (later rebranded as Trino due to copyright claims) that was created by the team of original presto authors who had departed from Facebook. After some extensive internal benchmarking, we chose the latter, since it has a better feature set and a much higher development pace. And since it wasn't available as a part of EMR, we ended up deploying PrestoSQL in the Kubernetes cluster, which allowed us to scale the cluster up and down better and faster depending on the current workload.

​It's not a perfect tool, though — a fast development pace means that sometimes you find things broken, but that's OK).​

Presto: SQL on Everything