Delivering value is the ultimate goal in product development, growth hacking, and data science. While there are different ways to get there, experimentation is the fastest, surest path to understanding what works with your users at scale. Many teams are so focussed on product outputs that they forget to create a technology foundation for driving better outcomes. The steps below will help you build a smarter tech stack for testing and learning what works with customers using progressive delivery and experimentation.
If you watched this year’s Test and Learn and Developer Summit, you may have heard Optimizely’s Chief Technology Officer, Lawrence Bruhmuller’s talk about the future of software development. He explained how feature flagging and experimentation are two sides of the same coin. The former allows software development teams to validate quality and performance in production, targeting key segments of your user base, and adapting based on real user impact. Experimentation is a natural extension of progressive delivery. It lets you validate new features and changes as A/B tests, and iterate on functionality and UX that confirms you are building the right things, the right way. Your key metrics can be conversion or engagement measurements. Success can also be measured by under the hood metrics like latency, uptime, or error rates.
Together, progressive delivery and experimentation form a single motion. When developers write code to support this motion, product managers and business stakeholders can question, iterate, and validate to deliver better customer experiences. Meanwhile, data scientists can rely on statistical rigor and machine learning to yield trusted results. For a more holistic customer view, they can enhance internal data pipelines with experiment tracking.
Follow the steps below to build an architecture that fosters a ‘test and learn’ approach and drive better outcomes.
🔎 Step One: Define your goals for managing product delivery and gathering trusted insights.
The shortlist might look something like this:
- Test and iterate on ideas.
- Quickly identify the highest value product opportunities.
- Reserve engineering time for the most significant development.
- Validate code quality and performance.
- Prevent negative user experiences.
- Introduce the right features to the right audiences, in the most impactful ways.
- Use data to understand our customers.
Consider how your goals map back to phases of your development process.
Optimizely’s VP of Product, Jon Noronha, used the innovation loop above to introduce the Test and Learn keynote speaker, Nicole Forsgren, VP of Research and Strategy at Github. While the ‘build, measure, learn’ pattern is typical in product development cycles, it’s essential to identify where you need to invest at each stage to meet your goals.
👷🏻♀️ Step Two: Evaluate where work needs to be done to achieve your goals listed in step one in your tech stack.
Where in your technology ecosystem could feature flags and experiments provide better results for your business?
The Product and Engineering team at HelloFresh shared a guide for retention experiments using Optimizely. The team explained how Optimizely’s flexible platform enables them to run retention experiments for extended periods to show long-term results. They use feature flagging and A/B testing on their website and work tirelessly to improve their iOS and Android apps. When they release new functionality, they do it gradually, as an experiment region by region. The outcome is a great user experience that leads to higher conversion, retention, and business margins.
Do gaps in your data pipeline limit your customer insights?
For modern product teams, using data to make decisions is table stakes. How customers react to those decisions is entirely dependent on the quality of the data itself. If your inputs are incomplete, inconsistent, or hard to take action on, then you probably lack a reliable view of your customers and how to deliver value.
Andy Yeo, Product Manager at Segment, spoke about customer empathy and customizations that scale. He showed how a Customer Data Platform (CDP) collects and streamlines inputs from all of your applications and channels to make the information highly actionable.
Integrate Segment with Optimizely as a destination by measuring your experiments against the unified metrics tracked in Segment. And you can implement Optimizely as a data source, automatically sending experiment events to Segment so you can track which customers see which A/B test variants.
You get a fuller picture of your customers when sending Optimizely events to our CDP partners, Segment, mParticle, and Tealium. Or, send data directly to any analytics provider/BI tool using our flexible integrations. Either option will provide the data pipeline to see how your experiments affect your downstream metrics like retention and lifetime value.
Could complementing your internal tools help you meet your goals?
Farfetch’s Principal Product Manager, Luís Trindade, is passionate about choosing the right metrics. He uses Optimizely Web for rapid prototyping and iteration to quantify and prove the value of his work. Farfetch has an internal platform for server-side testing. It uses Optimizely Full Stack to power parts of their experimentation platform, enabling the team to leverage more advanced experimentation features and measure business impact before committing to exploratory ideas.
When considering a partner to augment your internal platform, evaluate how easily you can integrate the systems. Optimizely’s Full Stack SDKs are open source (see GitHub repositories), and Rest API is flexible and developer-friendly.
Use this guide to understand the considerations and costs for when to build, buy, or complement a progressive delivery and experimentation solution for your organization.
💡 Step Three: Develop an innovation model on questioning, iterating, and proving value for every feature.
Forsgren explained how staying ahead of your competition is about being data-driven and unafraid to try new things. Technology must be in place for teams to quickly run A/B tests and get reliable results. But before a team gets the full advantage of ‘test and learn,’ a culture must exist to support the practices. Innovation happens when both the development principles and technology support creativity and experimentation.
Ensure you are building a ‘test and learn’ foundation in your features, work, and mental model by asking the following questions as part of your development plans.
- What is a good outcome for this product?
- Are there different iterations of the feature that should be configurable without a deployment?
- How can development be minimized to validate the approach?
- Which users should be the first to test the product?
- Which customer segments are likely to respond in unique ways?
- How can the product rollout be phased, managed, and iterated on outside of deployment cycles?
- What upfront investments will minimize risk and unexpected engineering requests?
By considering these questions in product development, you can develop your features with a balance of customer-centricity, focus on business value, resourcefulness, and efficiency.
Developing software with a smarter stack means that your product can be configured and tested with real users. Your ability to adapt quickly based on data-driven customer insights is critical for success in a dynamic world.
If you’re evaluating how to best serve your team’s needs for feature flagging and experimentation, download the guide to determine when to build, buy, or compliment.
4
1377
