Redux: Scaling LaunchDarkly From 4 to 200 Billion Feature Flags Daily

6,658
LaunchDarkly
Serving over 20 trillion feature flags daily to help software teams build better software, faster. LaunchDarkly helps eliminate risk for developers and operations teams from the software development cycle.

Written By John Kodumal, CTO and Co-Founder, LaunchDarkly


Background

LaunchDarkly is a feature management platform—we make it easy for software teams to adopt feature flags, helping them eliminate risk in their software development cycles. When we first wrote about our stack, we served about 4 billion feature flags a day. Last month, we averaged over 200 billion flags daily. To me, that's a mind-boggling number, and a testament to the degree to which we're able to change the way teams do software development. Some additional metrics:

  • Our global P99 flag update latency (the time it takes for a feature flag change on our dashboard to be reflected in your application) is under 500ms
  • Our primary Elasticsearch cluster indexes 175M+ docs / day
  • At daily peak, 1.5 million+ mobile devices and browsers and 500k+ servers are connected to our streaming APIs
  • Our event ingestion pipeline processes 40 billion events per day

We've scaled all our services through a process of gradual evolution, with an occasional bit of punctuated equilibrium. We've never re-written a service from scratch, nor have we ever had to completely re-architect any of our services (we did migrate one service from a SaaS provider to a homegrown; more on that later). In fact, from a high level, our stack is very similar to what we described in our earlier post:

  • A Go monolith that serves our REST API and UI (JS / React)
  • A Go microservice that powers our streaming API
  • An event ingestion / transformation pipeline implemented as a set of Go microservices

We use AWS as our cloud provider, and Fastly as our CDN.

Let's talk about some of the changes we've made to scale these systems.

Buy first, build if necessary

Over the past year, we've shifted our philosophy on managed services and have moved several critical parts of our infrastructure away from self-managed options. The most prominent was our shift away from HAProxy to AWS's managed application load balancers (ALBs). As we scaled, managing our HAProxy fleet became a larger and larger burden. We spent a significant amount of time tuning our configuration files and benchmarking different EC2 instance types to maximize throughput. Emerging needs like DDoS protection and auto scaling turned into large projects that we needed to schedule urgently. Instead of continuing this investment, we chose to shift to managed ALB instances. This was a large project, but it quickly paid for itself as we've nearly eliminated the time spent managing load balancers. We also gained DDoS protection and auto scaling "for free".

As we've evolved or added additional infrastructure to our stack, we've biased towards managed services:

  • Most new backing stores are Amazon RDS instances now. We do use self-managed PostgreSQL with TimescaleDB for time-series data—this is made HA with the use of Patroni and Consul.
  • We also use managed Elasticache instances instead of spinning up EC2 instances to run Redis workloads.
  • In our previous StackShare article, I wrote about a project to incorporate Kafka into our event ingestion pipeline. In keeping with our shift towards managed services, we shifted to Amazon's Kinesis instead of Kafka.

Managed services do have some drawbacks:

  • They're almost never cheaper (in raw dollars) than self-managed alternatives. Pricing is often more opaque, more variable, and hard to predict
  • Much less visibility into the operation, errors, and availability of the service
  • Vendor lock-in

Still, it's a false economy to measure the raw cost of a managed service to an unmanaged service—factor in your team's time and the math is usually pretty clear.

There is one notable case where we've moved from a managed SaaS solution to a homegrown. LaunchDarkly relies on a novel streaming architecture to push feature flag changes out in near real-time. Our SDKs create persistent outbound HTTPS connections to the LaunchDarkly streaming APIs. When you change a feature flag on your dashboard, that change is pushed out using the server-sent events (SSE) protocol. When we initially built our streaming service, we relied heavily on a third-party service, Fanout, to manage persistent connections. Fanout worked well for us, but over time we found that we could introduce domain-specific performance and cost optimizations if we built a custom service for our use case. We created a Go microservice that manages persistent connections and is heavily optimized for the unique workloads associated with feature flag delivery. We use NATS as a message broker to connect our REST API to a fleet of EC2 instances running this microservice. Each of these instances can manage over 50,000 concurrent SSE connections.

At scale, everything is a tight loop

Some of our analytics services receive tens of thousands of requests per second. One of the biggest things we've learned over the past year is that at this scale, there's almost no such thing as premature optimization. Because of the sheer volume of requests, every handler you write is effectively running in a tight loop. We found that to keep meeting our service level objectives and cost goals at scale, we had to do two things repeatedly:

  1. Profile aggressively to identify and address CPU and memory bottlenecks
  2. Apply a set of micro-patterns to handle specific workload

Profiling must be done periodically, as new bottlenecks will constantly emerge as traffic scales and old bottlenecks are eliminated. As an example, at one point, we found that the "front-door" microservice for our analytics pipeline was CPU-bound parsing JSON. We switched from Go's built-in encoding/json package to easyjson, which uses compile-time specialization to eliminate slow runtime reflection in JSON parsing.

We also identified a set of "micro-patterns" that we have extracted as self-contained libraries so they can be applied in appropriate contexts. Some examples:

  • Read coalescing—In a read-heavy workload, expensive calls to fetch data can be queued to await the first read—a kind of memoization. This pattern is encapsulated in Google's singleflight package
  • Write coalescing—The dual of read coalescing. In a write-heavy workload, where last write wins, writes can be queued and discarded in favor of the latest write attempt.
  • Multi-layer caching—In scenarios where an in-process, in-memory cache is necessary for performance, horizontal scaling can reduce cache hit rates. We make our fleet more resilient to this effect by employing multiple layers of caching—for example, backing an in-memory cache with a shared Redis cache before finally falling back to a slower persistent disk-backed store.

These simple patterns improved performance at scale and also helped us deal with bad traffic patterns like reconnection storms.

Get good at managing change

Scaling up isn't just about improving your services and architecture. It requires equal investment in people, processes and tools. One thing we really focused on the process and tools front is understanding change. Better visibility into changes being made to the service had a massively positive impact on service reliability. Here are a few things we did to improve visibility:

  • Internal changelog service: This service catalogues intentional changes being made to the system. This includes deploys, instance type changes, configuration changes, feature flag changes, and more. Anything that could potentially impact the service (either in a positive or negative way) is catalogued here. We couldn't find anything off the shelf here, so we built something ourselves.
  • COGS (cost of goods sold) log: Very similar to our changelog, but focused on price changes to our services. If we scale out a service, or change instance types, or make reserved instance reservations, we add an entry to this log. For us, this is just a Confluence page.
  • Observability / APM: We use a number of services to gain observability into what is happening to our service at runtime. We use a mix of Graphite / Grafana and Honeycomb.io to give us the observability we need. We're big fans of Honeycomb here.
  • Operational and release feature flags: We feature flag most changes using LaunchDarkly. Most new changes are protected by release flags (short-lived flags that are used to protect the initial rollout and rollback of a feature). We also create operational flags—which are long-lived flags that act as control switches to the application. Observability lets us understand change, and feature flags allow us to react to change to maintain availability or improve user experience.
  • Spinnaker / Armory: LaunchDarkly is almost a five year old company, and our methodology for deploying was state of the art... for 2014. We recently undertook a project to modernize the way we deploy our software, moving from Ansible-based deploy scripts that executed on our local machines, to using Spinnaker (along with Terraform and Packer) as the basis of our deployment system. We've been using Armory's enterprise Spinnaker offering to make this project a reality.

Like the sound of this stack? Learn more about LaunchDarkly.

LaunchDarkly
Serving over 20 trillion feature flags daily to help software teams build better software, faster. LaunchDarkly helps eliminate risk for developers and operations teams from the software development cycle.
Tools mentioned in article
Open jobs at LaunchDarkly
Platform Engineer
- US
<p data-renderer-start-pos="2137"><strong data-renderer-mark="true">About the Job:&nbsp;</strong></p> <p data-renderer-start-pos="2154">As a Platform Engineer, you will help us extend and scale LaunchDarkly's engineering infrastructure. You will build an internal development platform that brings joy to your end users - LaunchDarkly developers - and you are passionate about multiplying our engineering force through automation, orchestration, documentation, and productivity tools.</p> <p data-renderer-start-pos="2503">LaunchDarkly is a rapidly growing software company with a strong mission and vision carried out by a talented and diverse team of employees. Our goal is to help teams build better software, faster. You'll join a small team from companies like Atlassian, Intercom, and Twitter, and you'll have an immediate impact with our product and customers.</p> <h4>Responsibilities:</h4> <ul> <li> <p data-renderer-start-pos="2874">Collaborate with engineering teams to identify bottlenecks and areas for improvement in their development lifecycle, including Infrastructure as Code (IaC) templates for consistent provisioning.</p> </li> <li> <p data-renderer-start-pos="3072">Implement tools and processes to increase developer productivity and reduce time-to-market.</p> </li> <li> <p data-renderer-start-pos="3167">Automate infrastructure provisioning, scaling, and management to ensure efficiency and reliability.</p> </li> <li> <p data-renderer-start-pos="3270">Lead the design and implementation of our internal developer platform, empowering development teams to build and operate their services faster and more effectively.</p> </li> <li> <p data-renderer-start-pos="3438">Ensure the platform is scalable, secure, and aligned with best practices.</p> </li> <li> <p data-renderer-start-pos="3515">Troubleshoot and resolve delivery pipeline issues to ensure a reliable release process.</p> </li> <li> <p data-renderer-start-pos="3606">Keep up-to-date with platform engineering industry trends and emerging technologies.</p> </li> </ul> <h4>Qualifications:</h4> <ul> <li> <p data-renderer-start-pos="3713">You possess excellent communication and collaboration skills. You play well on a small, tight-knit team.</p> </li> <li> <p data-renderer-start-pos="3821">You are a self‐starter and problem solver, willing to solve hard problems and work independently when necessary. You identify potential problems and nip them in the bud before they surface.</p> </li> <li> <p data-renderer-start-pos="4014">You have a passion for reducing toil for yourself and your colleagues, and love solving problems with scripts and automation (e.g. with Python and Bash.)</p> </li> <li> <p data-renderer-start-pos="4171">You have successfully run large-scale production systems on public cloud infrastructure.</p> </li> <li> <p data-renderer-start-pos="4263">You have strong proficiency in building and maintaining CI/CD pipelines with tools like CircleCI and Spinnaker.</p> </li> <li> <p data-renderer-start-pos="4378">You are familiar with running systems with a service-oriented architecture.</p> </li> <li> <p data-renderer-start-pos="4457">You have written software in languages like Go or Python.</p> </li> <li> <p data-renderer-start-pos="4518">You have extensive experience codifying infrastructure with tools like Terraform.</p> </li> </ul> <div> <p><strong>Pay:</strong></p> <p>Target pay ranges based on Geographic Zones* for Level P3:</p> <ul> <li>Zone 1: San Francisco/Bay Area or New York City Metropolitan Area: $150,000-176,000**</li> <li>Zone 2: Boston, DC, Irvine, LA, Monterey, Santa Barbara, Santa Rosa, Seattle: $135,000-158,000**</li> <li>Zone 3: All other US locations: $127,000-150,000**</li> </ul> <p>Target pay ranges based on Geographic Zones* for Level P5:</p> <ul> <li>Zone 1: San Francisco/Bay Area or New York City Metropolitan Area: $200,000-235,000**</li> <li>Zone 2: Boston, DC, Irvine, LA, Monterey, Santa Barbara, Santa Rosa, Seattle: $180,000-212,000**</li> <li>Zone 3: All other US locations: $170,000 - $200,000**</li> </ul> <p data-renderer-start-pos="6572"><em data-renderer-mark="true">*Within the United States, our geographic pay zones are defined by counties surrounding major metropolitan areas.<br>**Restricted Stock Units (RSUs), health, vision, and dental insurance, and mental health benefits in addition to salary.</em></p> <p data-renderer-start-pos="6572"><em data-renderer-mark="true">LaunchDarkly operates from a place of high trust and transparency; we are happy to state the pay range for our open roles to best align with your needs. Exact compensation may vary based on skills, experience, degree level, and location.</em></p> </div> <h4><strong>About LaunchDarkly:</strong></h4> <p>Modern software delivery was supposed to be the foundation for a thriving digital business but reality has proven otherwise. Slow, inefficient development cycles, costly outages, and fragmented customer experiences are preventing developers from building their best software. The LaunchDarkly platform helps developers innovate on new features faster while protecting them with a safety valve to instantly rewind when things go wrong. Developers can target product experiences to any customer segment and maximize the business impact of every feature. And by gradually rolling out new application components, they escape nightmare "big-bang" technology migrations.&nbsp;</p> <p>The LaunchDarkly platform was built to guide engineers to the next frontier of DevOps by:</p> <ul> <li>Improving the velocity and stability of software releases, without the fear of end customer outages</li> <li>Delivering targeted experiences by easily personalizing features to customer cohorts</li> <li>Maximizing the business impact of every feature through the ability to experiment and optimize</li> <li>Coordinating the release and optimization of software to provide consistent experiences across mobile platforms and device types</li> <li>Improving the effectiveness and productivity of engineering teams, by providing insights into engineering cadence and stability</li> </ul> <p>At LaunchDarkly, we believe in the power of teams. We're building a team that is humble, open, collaborative, respectful and kind. We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, gender identity, sexual orientation, age, marital status, veteran status, or disability status.<br><br>One of our company values is 'Widen the Circle'. Which means we seek out diversity of perspectives to get better results. We understand everyone has their own unique talents and experiences. We encourage you to apply to this role even if you don’t think you meet 100% of the qualifications outlined above. We can find out together if it's the right match for your skillset.</p> <p class="p1">Learn more about the interview process &amp; what it’s like to work with <a class="css-tgpl01" href="https://www.gem.com/lp/landing-pages/launchdarkly-product-delivery" data-testid="link-with-safety" data-renderer-mark="true">LaunchDarkly's Product Delivery team</a>!</p> <p class="p1"><span class="s1"><strong>Do you need a disability accommodation?</strong></span></p> <p class="p1">Fill out this&nbsp;<a href="https://docs.google.com/forms/d/e/1FAIpQLSdYb_7upYMtdRVXzvXGHGfQw0pU2FNma-6Rwp-I6NjKm7SYNw/viewform"><span class="s2">accommodations request form</span></a>&nbsp;and someone from our People Operations team will contact you for assistance.&nbsp;</p>
Enterprise Solutions Engineer (Pre-Sa...
London, England, United Kingdom
<h4>About the Job:</h4> <p>As a Solutions Engineer, you will educate and guide prospects on the proper implementation of LaunchDarkly's SaaS product and Private Instances. You are passionate about trends and technologies involved in modern application development.&nbsp;You will be the technical voice during our sale and ensure our customers are comfortable with the way our systems work. You are passionate about the developer tools space and helping development teams eliminate risk and deliver value.</p> <p data-renderer-start-pos="1601">LaunchDarkly is a rapidly growing software company with a strong mission and vision carried out by a talented and diverse team of employees. Our goal is to help teams build better software, faster. Software powers the world and LaunchDarkly empowers all teams to deliver and control their software.</p> <p data-renderer-start-pos="1601">Please note this position is a Pre-Sales Engineer role, working alongside our existing SE team and providing key support and technical insight throughout the sales cycle.&nbsp;</p> <h4>Responsibilities:</h4> <ul> <li data-renderer-start-pos="1924">Evangelize and advise customers on the importance and different uses of feature flags and how to administer them</li> <li data-renderer-start-pos="2040">Presenting solutions to customer's challenges implementing feature flags across large monolith and microservice applications, large organizations, and different technology stacks</li> <li data-renderer-start-pos="2218">Become a domain expert on LaunchDarkly architecture</li> <li data-renderer-start-pos="2273">Demo LaunchDarkly product to technical and business audiences</li> <li data-renderer-start-pos="2338">Become a subject matter expert on LaunchDarkly and communicate our value and features to potential customers</li> <li data-renderer-start-pos="2450">Be the voice of the customer by translating, aggregating, and representing customer feedback to the Product and Engineering teams</li> </ul> <h4>Qualifications:</h4> <ul> <li data-renderer-start-pos="2608">4+ years of experience consulting with enterprise customers and large development teams</li> <li data-renderer-start-pos="2700">You led successful technical proof of concepts&nbsp;</li> <li data-renderer-start-pos="2751">Proven success in building strong customer relationships</li> <li data-renderer-start-pos="2811">Ability to learn and synthesize large amounts of information with little context</li> <li data-renderer-start-pos="2895">Effective communicator with the ability to simplify complex technical concepts</li> <li data-renderer-start-pos="2977">A self‐starter and problem solver, willing to take on hard problems and work independently when necessary.</li> <li data-renderer-start-pos="3116">Experience working with teams that underwent development process transformation</li> <li data-renderer-start-pos="3199">Familiarity with at least one of our supported languages: Java, .NET, GO, JS, Python, PHP, Node, Ruby, Rails, iOS, or Android</li> <li data-renderer-start-pos="3328">Experience with data persistence technologies like Varnish or Redis</li> </ul> <p><strong>Specific Requirements:</strong></p> <ul> <li>Must be able and willing to travel as needed</li> </ul> <h4><strong>About LaunchDarkly:</strong></h4> <p>Modern software delivery was supposed to be the foundation for a thriving digital business but reality has proven otherwise. Slow, inefficient development cycles, costly outages, and fragmented customer experiences are preventing developers from building their best software. The LaunchDarkly platform helps developers innovate on new features faster while protecting them with a safety valve to instantly rewind when things go wrong. Developers can target product experiences to any customer segment and maximize the business impact of every feature. And by gradually rolling out new application components, they escape nightmare "big-bang" technology migrations.&nbsp;</p> <p>The LaunchDarkly platform was built to guide engineers to the next frontier of DevOps by:</p> <ul> <li>Improving the velocity and stability of software releases, without the fear of end customer outages</li> <li>Delivering targeted experiences by easily personalizing features to customer cohorts</li> <li>Maximizing the business impact of every feature through the ability to experiment and optimize</li> <li>Coordinating the release and optimization of software to provide consistent experiences across mobile platforms and device types</li> <li>Improving the effectiveness and productivity of engineering teams, by providing insights into engineering cadence and stability</li> </ul> <p>At LaunchDarkly, we believe in the power of teams. We're building a team that is humble, open, collaborative, respectful and kind. We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, gender identity, sexual orientation, age, marital status, veteran status, or disability status.<br><br>One of our company values is 'Widen the Circle'. Which means we seek out diversity of perspectives to get best results. We understand everyone has their own unique talents and experiences. We encourage you to apply to this role even if you don’t think you meet 100% of the qualifications outlined above. We can find out together if it's the right match for your skillset.</p> <p class="p1"><span class="s1"><strong>Do you need a disability accommodation?</strong></span></p> <p class="p1">Fill out this&nbsp;<a href="https://docs.google.com/forms/d/e/1FAIpQLSdYb_7upYMtdRVXzvXGHGfQw0pU2FNma-6Rwp-I6NjKm7SYNw/viewform"><span class="s2">accommodations request form</span></a>&nbsp;and someone from our People Operations team will contact you for assistance.&nbsp;</p> <p>&nbsp;</p>
Senior Analytics Engineer
Oakland, California, United States
<h4><strong>About the Job:<br></strong></h4> <p>LaunchDarkly is seeking an experienced Analytics Engineer to join our data team. You will work with other Data Engineers, Analytics Engineers, and stakeholders across the company to deliver data insights that create material impacts to our business.</p> <h4>Responsibilities:</h4> <ul class="ak-ul" data-indent-level="1"> <li> <p data-renderer-start-pos="2997">Develop data-driven solutions to critical business problems by leveraging our data warehouse, analytics tools, and scripting languages.</p> </li> <li> <p data-renderer-start-pos="3136">Partner with stakeholders throughout the company to drive improved business results in their areas by providing creative technical solutions to their most pressing business challenges.</p> </li> <li> <p data-renderer-start-pos="3325">Translate quantitative analyses and findings into clear visuals, written documentation, and other data assets for non-technical end users.</p> </li> <li> <p data-renderer-start-pos="3468">Provide suggestions for improving the reliability, ease of use, and performance of the Data Team’s technical infrastructure.</p> </li> <li> <p data-renderer-start-pos="3596">Form strong relationships with your team and refine our best practices e.g. naming convention, data modeling, and data quality testing.</p> </li> <li> <p data-renderer-start-pos="3735">Enable users across the organization to self-serve requests using our explorable data sets</p> </li> </ul> <h4>Qualifications:</h4> <ul> <li> <p data-renderer-start-pos="3848">5+ years of professional experience as an Analytics Engineer, Data Scientist, or similar.</p> </li> <li> <p data-renderer-start-pos="3941">A Bachelor’s degree in a technical field (Computer Science, Physics, Math, Economics, etc)</p> </li> <li> <p data-renderer-start-pos="4036">An ability to balance humility with rigorous standards for work quality</p> </li> <li> <p data-renderer-start-pos="4111">A curious and growth-oriented mindset towards learning and teaching.</p> </li> <li> <p data-renderer-start-pos="4183">Ability to understand, tackle, and communicate problems from both technical and business perspectives</p> </li> <li> <p data-renderer-start-pos="4288">Experience with data modeling and warehousing concepts</p> </li> <li> <p data-renderer-start-pos="4346">Proficient in Snowflake (SQL), Python, and visualization tools</p> </li> <li> <p data-renderer-start-pos="4412">Proficient with advanced statistical techniques</p> </li> <li> <p data-renderer-start-pos="4463">Comfortable working in DBT and AWS Data Tools.</p> </li> </ul> <p><strong>Pay:</strong></p> <p data-renderer-start-pos="6395"><em data-renderer-mark="true">Target pay range for a Level P4 in San Francisco/Bay Area: </em><em>$173,000 - $204,000*</em></p> <p data-renderer-start-pos="6451"><em data-renderer-mark="true">*Restricted Stock Units (RSUs), health, vision, and dental insurance, and mental health benefits in addition to salary.</em></p> <p data-renderer-start-pos="6572"><em data-renderer-mark="true">LaunchDarkly operates from a place of high trust and transparency; we are happy to state the pay range for our open roles to best align with your needs. Exact compensation may vary based on skills, experience, degree level, and location.</em></p> <h4><strong>About LaunchDarkly:</strong></h4> <p>Modern software delivery was supposed to be the foundation for a thriving digital business but reality has proven otherwise. Slow, inefficient development cycles, costly outages, and fragmented customer experiences are preventing developers from building their best software. The LaunchDarkly platform helps developers innovate on new features faster while protecting them with a safety valve to instantly rewind when things go wrong. Developers can target product experiences to any customer segment and maximize the business impact of every feature. And by gradually rolling out new application components, they escape nightmare "big-bang" technology migrations.&nbsp;</p> <p>The LaunchDarkly platform was built to guide engineers to the next frontier of DevOps by:</p> <ul> <li>Improving the velocity and stability of software releases, without the fear of end customer outages</li> <li>Delivering targeted experiences by easily personalizing features to customer cohorts</li> <li>Maximizing the business impact of every feature through the ability to experiment and optimize</li> <li>Coordinating the release and optimization of software to provide consistent experiences across mobile platforms and device types</li> <li>Improving the effectiveness and productivity of engineering teams, by providing insights into engineering cadence and stability</li> </ul> <p>At LaunchDarkly, we believe in the power of teams. We're building a team that is humble, open, collaborative, respectful and kind. We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, gender identity, sexual orientation, age, marital status, veteran status, or disability status.<br><br>One of our company values is 'Widen the Circle'. Which means we seek out diversity of perspectives to get better results. We understand everyone has their own unique talents and experiences. We encourage you to apply to this role even if you don’t think you meet 100% of the qualifications outlined above. We can find out together if it's the right match for your skillset.</p> <p class="p1"><span class="s1"><strong>Do you need a disability accommodation?</strong></span></p> <p class="p1">Fill out this&nbsp;<a href="https://docs.google.com/forms/d/e/1FAIpQLSdYb_7upYMtdRVXzvXGHGfQw0pU2FNma-6Rwp-I6NjKm7SYNw/viewform"><span class="s2">accommodations request form</span></a>&nbsp;and someone from our People Operations team will contact you for assistance.&nbsp;</p>
Backend Engineer, Data Pipeline
Oakland, California, United States
<p data-renderer-start-pos="1895"><strong data-renderer-mark="true">About the Job:</strong></p> <p data-renderer-start-pos="1911">We are currently looking for a talented Software Engineer to join our Experimentation Product team. You will help us architect and write fast, reliable, and scalable data processing tools to process data from our thousands of customers and their hundreds of millions of users around the world. We're looking for someone who knows what it takes to deliver value to customers and takes pride in the quality of their work.</p> <p data-renderer-start-pos="2332">Our data platform processes events from over twenty billion feature flag evaluations, and billions of experimentation metrics events daily. The core technologies we use daily include Golang, Scala, Kinesis, and Flink. As part of our you-build-it-you-run-it culture, all developers may be responsible for supporting applications in production, including on-call.</p> <p data-renderer-start-pos="2695">Your work will directly reflect on the customer experience to allow them to run the most effective experiments possible.</p> <h4 id="Responsibilities:" data-renderer-start-pos="2821"><strong data-renderer-mark="true">Responsibilities:</strong></h4> <ul class="ak-ul" data-indent-level="1"> <li> <p data-renderer-start-pos="2842">Working with a modern data stack and open source frameworks</p> </li> <li> <p data-renderer-start-pos="2905">Build and expand our data platform, pipeline, and services</p> </li> <li> <p data-renderer-start-pos="2967">Design data pipelines for new products and evolving customer needs</p> </li> <li> <p data-renderer-start-pos="3037">Collaborate with product team to influence and deliver user-facing features</p> </li> <li> <p data-renderer-start-pos="3116">Monitor and improve data pipeline performance for continuous improvement</p> </li> <li> <p data-renderer-start-pos="3192">Champion engineering standards, tooling, and processes</p> </li> </ul> <p data-renderer-start-pos="3250"><strong data-renderer-mark="true">Qualifications:</strong></p> <ul> <li data-renderer-start-pos="3269">Proven experience and fluency in a JVM language or Golang</li> <li> <p data-renderer-start-pos="3330">Experience building data platforms (e.g. using Flink, Kafka, DataFlow, Hadoop, Spark)</p> </li> <li> <p data-renderer-start-pos="3419">Experience with data pipelines, distributed systems, and large-scale data processing</p> </li> <li> <p data-renderer-start-pos="3507">Strong communication skills, a positive attitude, and empathy</p> </li> <li> <p data-renderer-start-pos="3572">You hold yourself and others to a high bar when working with data systems</p> </li> <li> <p data-renderer-start-pos="3649">You value high code quality, automated testing, and other engineering best practices</p> </li> <li> <p data-renderer-start-pos="3737">Experience with AWS</p> </li> <li> <p data-renderer-start-pos="3760">Experience with experimentation systems / products</p> </li> <li> <p data-renderer-start-pos="3814">Working knowledge of ETL and a query language</p> </li> </ul> <div> <p><strong>Pay:</strong></p> <p data-renderer-start-pos="3872"><em data-renderer-mark="true">Target pay range for a Level P5 in San Francisco/Bay Area: $200,000-$235,000*</em></p> <p data-renderer-start-pos="6451"><em data-renderer-mark="true">*Restricted Stock Units (RSUs), health, vision, and dental insurance, and mental health benefits in addition to salary.</em></p> <p data-renderer-start-pos="6572"><em data-renderer-mark="true">LaunchDarkly operates from a place of high trust and transparency; we are happy to state the pay range for our open roles to best align with your needs. Exact compensation may vary based on skills, experience, degree level, and location.</em></p> </div> <h4><strong>About LaunchDarkly:</strong></h4> <p>Modern software delivery was supposed to be the foundation for a thriving digital business but reality has proven otherwise. Slow, inefficient development cycles, costly outages, and fragmented customer experiences are preventing developers from building their best software. The LaunchDarkly platform helps developers innovate on new features faster while protecting them with a safety valve to instantly rewind when things go wrong. Developers can target product experiences to any customer segment and maximize the business impact of every feature. And by gradually rolling out new application components, they escape nightmare "big-bang" technology migrations.&nbsp;</p> <p>The LaunchDarkly platform was built to guide engineers to the next frontier of DevOps by:</p> <ul> <li>Improving the velocity and stability of software releases, without the fear of end customer outages</li> <li>Delivering targeted experiences by easily personalizing features to customer cohorts</li> <li>Maximizing the business impact of every feature through the ability to experiment and optimize</li> <li>Coordinating the release and optimization of software to provide consistent experiences across mobile platforms and device types</li> <li>Improving the effectiveness and productivity of engineering teams, by providing insights into engineering cadence and stability</li> </ul> <p>At LaunchDarkly, we believe in the power of teams. We're building a team that is humble, open, collaborative, respectful and kind. We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, gender identity, sexual orientation, age, marital status, veteran status, or disability status.<br><br>One of our company values is 'Widen the Circle'. Which means we seek out diversity of perspectives to get better results. We understand everyone has their own unique talents and experiences. We encourage you to apply to this role even if you don’t think you meet 100% of the qualifications outlined above. We can find out together if it's the right match for your skillset.</p> <p class="p1">Learn more about the interview process &amp; what it’s like to work with <a class="css-tgpl01" href="https://www.gem.com/lp/landing-pages/launchdarkly-product-delivery" data-testid="link-with-safety" data-renderer-mark="true">LaunchDarkly's Product Delivery team</a>!</p> <p class="p1"><span class="s1"><strong>Do you need a disability accommodation?</strong></span></p> <p class="p1">Fill out this&nbsp;<a href="https://docs.google.com/forms/d/e/1FAIpQLSdYb_7upYMtdRVXzvXGHGfQw0pU2FNma-6Rwp-I6NjKm7SYNw/viewform"><span class="s2">accommodations request form</span></a>&nbsp;and someone from our People Operations team will contact you for assistance.&nbsp;</p>
Verified by
Software and Product Engineer
Software Engineer
Computer Science
Physics
Director Marketing
Software Engineer
Engineering Manager
Software Engineer
Software Engineeer
VP of Product and Engineering
Engineering Lead
Software Engineer
Special Circumstances
Demand Program Manager
You may also like