How to Disable Code: The Developer’s Production Kill Switch

667
CloudBees
CloudBees, the enterprise software delivery company, provides the industry’s leading DevOps technology platform. CloudBees enables developers to focus on what they do best: Build stuff that matters, while providing peace of mind to management with powerful risk mitigation, compliance and governance tools.

The following is a guest post written by Carlos Schults.

Being able to disable code in production is a power that many developers aren’t aware of. And that’s a shame. The ability to switch off some portions—or even complete features—of the codebase can dramatically improve the software development process by allowing best practices that can shorten feedback cycles and increase the overall quality.

So, that’s what this post will cover: the mechanisms you can use to perform this switching off, why they’re useful and how to get started. Let’s dig in.

Why Would You Want to Disable Code?

Before we take a deep dive into feature flags, explaining what they are and how they’re implemented, you might be asking: Why would people want to switch off some parts of their codebase? What’s the benefit of doing that?

To answer these questions, we need to go back in time to take a look at how software was developed a couple of decades ago. Time for a history lesson!

The Dark Ages: Integration Hell

Historically, integration has been one of the toughest challenges for teams trying to develop software together.

Picture several teams inside an organization, working separately for several months, each one developing its own feature. While the teams were working in complete isolation, their versions of the application were evolving in different directions. Now they need to converge again into a single, non conflicting version. This is a Herculean task.

That’s what “integration hell” means: the struggle to merge versions of the same application that have been allowed to diverge for too long.

Enter the Solution: Continuous Integration

“If it hurts, do it more often.” What this saying means is that there are problems we postpone solving because doing so is hard. What you often find with these kinds of problems is that solving them more frequently, before they accumulate, is way less painful—or even trivial.

So, how can you make integrations less painful? Integrate more often.

That’s continuous integration (CI) in a nutshell: Have your developers integrate their work with a public shared repository, at the very least once a day. Have a server trigger a build and run the automated test suite every time someone integrates their work. That way, if there are problems, they’re exposed sooner rather than later.

How to Handle Partially Completed Features

One challenge that many teams struggle with in CI is how to deal with features that aren’t complete. If developers are merging their code to the mainline, that means that any developments that take more than one day to complete will have to be split into several parts.

How can you avoid the customer accessing unfinished functionality? There are some trivial scenarios with similarly trivial solutions, but harder scenarios call for a different approach: the ability to switch off a part of the code completely.

Feature Flags to the Rescue

Defining Feature Flags

There are many names for the mechanisms that allow developers to switch a portion of their code off and on. Some call them “feature toggles” or “kill switches.” But “feature flags” is the most popular name, so that’s what we’ll use for the remainder of this post. So, what are feature flags?

Put simply, feature flags are techniques that allow teams to change the behavior of an application without modifying the code. In general, flags are used to prevent users from accessing and using the changes introduced by some piece of code, because they’re not adequate for production yet for a number of reasons.

Disable Code: What Are the Use Cases?

We’ll now cover some of the most common use cases for disabling code in production.

Switching Off Unfinished Features

As you’ve seen, one of the main use cases for feature flags is preventing users from accessing features that aren’t ready for use yet.

That way, programmers developing features that are more complex and take a longer time to complete aren’t prevented from integrating their work often and benefiting from it.

Enabling A/B Testing

The adoption of feature flags enables the use of several valuable practices in the software development process, one of which is A/B testing.

A/B testing is a user experience research technique that consists of comparing two versions of a website or application to decide which one to keep. It entails randomly splitting users into two groups, A and B, and then delivering a different version of the application to each group. One group might receive the current production version, which we call the “control,” whereas the second group would receive the candidate for the new version, called the “treatment.”

The testers then monitor the behavior of both groups and determine which of the versions achieved better results.

Feature flags are a practical way to enable A/B testing because they allow you to quickly and conveniently change between the control and treatment versions of your application.

Enabling Canary Releases

If you deliver the new version of your app to your entire userbase at once, 100 percent of your users will be impacted if the release is bad in some way. What if you could gradually roll out the new version instead? You’d first deploy to a small subset of users, monitoring that group to detect issues. If something went wrong, you could roll it back. If everything looked fine, you could then gradually release the version for larger groups. That’s a canary release in a nutshell. It’s another powerful technique that feature flags might help with.

Customizing Features According to Users’ Preferences

It’s not uncommon to have to customize your application according to the needs of specific users, and there are several ways in which software teams can accomplish that—some more efficient, and others less so (companies that create separate branches or entire repositories for each client come to mind).

This is another area where feature flags could help, allowing teams to dynamically switch between different versions of the same functionality.

Disable Code in Production 101

How do you go about disabling code? That’s what we’re going to see now, in three increasingly sophisticated phases.

First Stage: The Most Basic Approach

We start with an approach that’s so primitive, it maybe shouldn’t be considered a feature flag at all. Consider the pseudocode below:

calculateAdditionalWorkHours(Employee employee, Date start, Date end) {     
    // return calculateAdditionalWorkHoursSameOldWay(employee, start, end);
    return calculateAdditionalWorkHoursImproved(employee, start, end); 
    }

In the code above, we're just commenting out the old version of some method and replacing it with a new version. When we want the older version to be used, we just do the opposite. (Well, I said it was primitive.) This approach lacks one of the most fundamental properties of a feature flag—the ability to change how the application behaves without changing its code.

However, it plants the seed for more sophisticated approaches.

Second Stage: Taking the Decision Out of the Code

The previous approach didn't allow developers to select the desired version of the feature without changing the code. Fortunately, that's not so hard to do. First, we introduce a logical variable to determine which version we're going to use:

calculateAdditionalWorkHours(Employee employee, Date start, Date end) {

    var result = useNewCalculation
        ? calculateAdditionalWorkHoursImproved(employee, start, end)
        : calculateAdditionalWorkHoursSameOldWay(employee, start, end);

    return result;
}

Then, we use some mechanism to be able to assign the value to the variable from an external source. We could use a configuration file:

var useNewCalculation = config[newCalculation];

Passing arguments to the application might be another option. What matters is that we now have the ability to modify how the app behaves from the outside, which is a great step toward "true" feature flagging.

Keep in mind that the code examples you see are all pseudocode. Using your favorite programming language, there's nothing stopping you from starting with this approach and taking it up a notch. You could, for instance, use classes to represent the features and design patterns (e.g., factories) to avoid if statements.

Stage 3: Full-Fledged Feature Flag Management

The previous approach might be enough when your application has only a small number of flags. But as that number grows, things start to become messy.

First, you have the issue of technical debt. Manually implemented feature flags can create terribly confusing conditional flows in your codebase. That only grows worse with new flags being introduced each day. Additionally, they might make the code harder to understand and navigate, especially for more junior developers, which is an invitation for bugs.

Another problem is that as the number of flags grows, it becomes more and more common to forget to delete old, obsolete ones.

The main problem of a homegrown approach is that it doesn't give you an easy way to see and manage all of your flags at once. That's why our third and final stage is a single piece of advice: Instead of rolling out your own feature flags approach, adopt a third-party feature flag management system.

Feature Flags Are a CI/CD Enabler

We've covered the mechanisms developers can use to disable portions of their codebase in production without having to touch the code. This capability is powerful and enables techniques such as A/B testing and canary releases, which are all hallmarks of a modern, agile-based software development process.

The names for the techniques might vary—feature flags, feature toggles, feature flipper, and so on. The way in which the techniques are implemented can also vary—from a humble if statement to sophisticated cloud-based solutions.

But no matter what you call them, you can't overstate the benefit these mechanisms offer. They're an enabler of Continuous Integration, which is essential for any modern software organization that wants to stay afloat.

CloudBees
CloudBees, the enterprise software delivery company, provides the industry’s leading DevOps technology platform. CloudBees enables developers to focus on what they do best: Build stuff that matters, while providing peace of mind to management with powerful risk mitigation, compliance and governance tools.
Tools mentioned in article
Open jobs at CloudBees
Lead DevOps Engineer
US East or EMEA

Our Customers Develop Software at the Speed of Ideas

CloudBees is powering the continuous economy by offering the world’s first end-to-end continuous software delivery management system (SDM). For millions of developers and product teams driving innovation for businesses large or small, SDM builds on continuous integration (CI) and continuous delivery (CD) to enable all functions and teams within and around the software delivery organization to best work together to amplify value creation.
 
CloudBees is the continuous integration (CI), continuous delivery (CD) and application release automation (ARA) powerhouse built from the commercial success of its products and its open source leadership as the largest contributor to Jenkins and a founding member of the Continuous Delivery Foundation (CDF). With a globally distributed workforce of more than 500 employees, the company reflects the global nature of the DevOps movement. We believe in walking the talk! From startups with full-stack developers practicing NoOps to large Fortune 100 companies, CloudBees enables all software-driven organizations to intelligently deploy the right capabilities at the right time.
  
Over 3,500 of the world’s best known brands and over 50% of the Fortune 500, invest in CloudBees because of its ability to work across any cloud, in any development environment and to balance corporate governance and control with developer flexibility and freedom.
 
CloudBees is home to the world’s leading DevOps experts helping thousands of companies harness the power of “continuous everything” and putting them on the fastest path from great idea, to great software, to great business value.

WHO WE ARE

The team you will be working on is responsible for making the CloudBees CI releases happen, using CloudBees CI, in a predictable and uneventful way.

Our team aims at being a catalyzer rather than a gatekeeper. For this, we provide and continuously improve automation and tooling so developers can self serve and we deliver value iteratively to customers in a fast and safe manner.

WHAT YOU'LL DO

  • Work in a rapidly iterating team in an agile environment to provide incremental features to the internal (software engineering team) and external customers 
  • Support critical applications and infrastructure, able to troubleshoot independently and create novel solutions
  • Work closely with related teams in the company to bring a culture of automation and self-service, in an inclusive and thoughtful manner
  • Streamline software development by implementing reusable tooling for CI/CD pipelines and automating manual processes and associated developer tooling
  • Participate in code review process, challenging others to be excellent
  • Mentor more junior team members
  • Continuously assess our pipelines and drive changes that will improve cycle times, decrease failure rates, reduce recovery times and remove complexity
  • Incorporate developer requests and suggestions in the design and improvement of our pipelines

WHO YOU ARE

  • BS in Engineering, Computer Science or related fields, or equivalent experience
  • Expert in Jenkins
  • 5+ years of software engineering and related experience in implementing devops practices.
  • Strong experience in implementing end-to-end devops pipelines, especially pipeline creation/management for CI/CD, continuous testing, deployment automation, leveraging pipeline automation, and custom job creation
    • Nice to have: experience orchestrating cloud resources with Terraform
  • Docker / container orchestration experience
  • Experience working in one of the various cloud providers – AWS or GCP
  • Strong interpersonal & leadership skills, and ability to deliver impactful changes that largely go beyond the team’s boundaries
  • Java language knowledge is a plus

WHAT YOU'LL GET

  • Highly competitive benefits and vacation package
  • Ability to work for one of the fastest growing companies with some of the most talented people in the industry
  • Team outings
  • Fun, Hardworking, and Casual Environment
  • Endless Growth Opportunities

At CloudBees, we truly believe that the more diverse we are, the better we serve our customers.  A global community like Jenkins demands a global focus from CloudBees. Organizations with greater diversity—gender, racial, ethnic, and global—are stronger partners to their customers.  Whether by creating more innovative products, or better understanding our worldwide customers, or establishing a stronger cross-section of cultural leadership skills, diversity strengthens all aspects of the CloudBees organization.

In the technology industry, diversity creates a competitive advantage.  CloudBees customers demand technologies from us that solve their software development, and therefore their business problems, so that they can better serve their own customers.  CloudBees attributes much of its success to its worldwide work force and commitment to global diversity, which opens our proprietary software to innovative ideas from anywhere. Along the way, we have witnessed firsthand how employees, partners, and customers with diverse perspectives and experiences contribute to creative problem solving and better solutions for our customers and their businesses.

Senior DevOps Consultant - Profession...
EMEA

OUR CUSTOMERS DEVELOP SOFTWARE AT THE SPEED OF IDEAS

CloudBees is the continuous integration (CI), continuous delivery (CD) and application release automation (ARA) powerhouse built from the commercial success of its products and a founding member of the Continuous Delivery Foundation (CDF). With a globally distributed workforce of more than 450 employees, the company reflects the global nature of the DevOps movement. We believe in walking the talk! From startups with full-stack developers practicing NoOps to large Fortune 100 companies, CloudBees enables all software-driven organizations to intelligently deploy the right capabilities at the right time.

Over 3,500 of the world’s best known brands and over 50% of the Fortune 500, invest in CloudBees because of its ability to work across any cloud, in any development environment and to balance corporate governance and control with developer flexibility and freedom.

CloudBees is the home for leading DevOps experts helping thousands of companies harness the power of “continuous everything” and putting them on the fastest path from great ideas, to great software, to great business value.

As a Senior Consultant, you will be working with our clients on how to overcome problems and inefficiencies with their software development and deployment processes. You will leverage your broad experience, troubleshooting abilities and architectural knowledge to implement the best approach to get the  best value from their investment in CloudBees technology. In the implementation process you will lead, architect and execute based on CloudBees proven delivery methodology. 

WHAT YOU’LL DO

  • Provide expert advice on all things related to CI, CD, and Release Management
  • Assess and make technical recommendations for customers to reach their DevOps targets
  • Help customers implement CloudBees products to enable Continuous Integration, Continuous Delivery and Release Management capabilities
  • Provide technical thought leadership for organizations and expert advice related to software design, architecture, and tooling as it relates to improving software delivery across the enterprise
  • Implement CloudBees automation and tracking tooling to help organizations gain true enterprise wide visibility and automation
  • Implement workflows and capabilities to drive towards everything as code

WHAT THE ROLE REQUIRES

  • Experience in fast-moving automated release environments using a variety of tools across configuration management, infrastructure orchestration, and IT Service Management
  • Direct experience in release engineering
  • Deep understanding of enterprise application design, dependencies, as well as modern and legacy applications architecture
  • Knowledge of Enterprise Architecture and the compliance and security concerns related to software development and deployment
  • Understanding of Release Management,  Continuous Integration and Continuous Delivery ecosystems and requirements
  • An understanding of Value Stream Management and process design
  • Practical experience with REST API, SOAP, Perl, Python, Groovy
  • Experience with Release Orchestration and Deployment Automation technology
  • Understanding of Agile development methods and how organizations adopt and implement these to support their end-to-end software development lifecycle processes
  • Strong written and verbal communication skills
  • Proven experience of Cloud providers such as AWS,Google and and Azure
  • Experience with of containers and Kubernetes implementations on major cloud providers and Openshift
  • Experience of training or mentoring others
  • Strong analytical and problem-solving skills
  • Travel- 75% (includes both customer and internal related travel)

WHAT YOU’LL GET

  • Highly competitive benefits and vacation package
  • Ability to work for one of the fastest growing companies with some of the most talented people in the industry
  • Team outings
  • Fun, Hardworking, and Casual Environment
  • Endless Growth Opportunities

At CloudBees, we truly believe that the more diverse we are, the better we serve our customers. A global community like Jenkins demands a global focus from CloudBees. Organizations with greater diversity—gender, racial, ethnic, and global—are stronger partners to their customers. Whether by creating more innovative products, or better understanding our worldwide customers, or establishing a stronger cross-section of cultural leadership skills, diversity strengthens all aspects of the CloudBees organization.

In the technology industry, diversity creates a competitive advantage. CloudBees customers demand technologies from us that solve their software development, and therefore their business problems, so that they can better serve their own customers. CloudBees attributes much of its success to its worldwide work force and commitment to global diversity, which opens our proprietary software to innovative ideas from anywhere. Along the way, we have witnessed firsthand how employees, partners, and customers with diverse perspectives and experiences contribute to creative problem solving and better solutions for our customers and their businesses.

 

Senior DevOps Consultant-Professional...
US

OUR CUSTOMERS DEVELOP SOFTWARE AT THE SPEED OF IDEAS

CloudBees is the continuous integration (CI), continuous delivery (CD) and application release automation (ARA) powerhouse built from the commercial success of its products and a founding member of the Continuous Delivery Foundation (CDF). With a globally distributed workforce of more than 450 employees, the company reflects the global nature of the DevOps movement. We believe in walking the talk! From startups with full-stack developers practicing NoOps to large Fortune 100 companies, CloudBees enables all software-driven organizations to intelligently deploy the right capabilities at the right time.

Over 3,500 of the world’s best known brands and over 50% of the Fortune 500, invest in CloudBees because of its ability to work across any cloud, in any development environment and to balance corporate governance and control with developer flexibility and freedom.

CloudBees is the home for leading DevOps experts helping thousands of companies harness the power of “continuous everything” and putting them on the fastest path from great ideas, to great software, to great business value.

As a Senior Consultant, you will be working with our clients on how to overcome problems and inefficiencies with their software development and deployment processes. You will leverage your broad experience, troubleshooting abilities and architectural knowledge to implement the best approach to get the  best value from their investment in CloudBees technology. In the implementation process you will lead, architect and execute based on CloudBees proven delivery methodology. 

WHAT YOU’LL DO

  • Provide expert advice on all things related to CI, CD, and Release Management
  • Assess and make technical recommendations for customers to reach their DevOps targets
  • Help customers implement CloudBees products to enable Continuous Integration, Continuous Delivery and Release Management capabilities
  • Provide technical thought leadership for organizations and expert advice related to software design, architecture, and tooling as it relates to improving software delivery across the enterprise
  • Implement CloudBees automation and tracking tooling to help organizations gain true enterprise wide visibility and automation
  • Implement workflows and capabilities to drive towards everything as code

WHAT THE ROLE REQUIRES

  • Experience in fast-moving automated release environments using a variety of tools across configuration management, infrastructure orchestration, and IT Service Management
  • Direct experience in release engineering
  • Deep understanding of enterprise application design, dependencies, as well as modern and legacy applications architecture
  • Knowledge of Enterprise Architecture and the compliance and security concerns related to software development and deployment
  • Understanding of Release Management,  Continuous Integration and Continuous Delivery ecosystems and requirements
  • An understanding of Value Stream Management and process design
  • Practical experience with REST API, SOAP, Perl, Python, Groovy
  • Experience with Release Orchestration and Deployment Automation technology
  • Understanding of Agile development methods and how organizations adopt and implement these to support their end-to-end software development lifecycle processes
  • Strong written and verbal communication skills
  • Proven experience of Cloud providers such as AWS,Google and and Azure
  • Experience with of containers and Kubernetes implementations on major cloud providers and Openshift
  • Experience of training or mentoring others
  • Strong analytical and problem-solving skills
  • Travel- 75% (includes both customer and internal related travel)

WHAT YOU’LL GET

  • Highly competitive benefits and vacation package
  • Ability to work for one of the fastest growing companies with some of the most talented people in the industry
  • Team outings
  • Fun, Hardworking, and Casual Environment
  • Endless Growth Opportunities

At CloudBees, we truly believe that the more diverse we are, the better we serve our customers. A global community like Jenkins demands a global focus from CloudBees. Organizations with greater diversity—gender, racial, ethnic, and global—are stronger partners to their customers. Whether by creating more innovative products, or better understanding our worldwide customers, or establishing a stronger cross-section of cultural leadership skills, diversity strengthens all aspects of the CloudBees organization.

In the technology industry, diversity creates a competitive advantage. CloudBees customers demand technologies from us that solve their software development, and therefore their business problems, so that they can better serve their own customers. CloudBees attributes much of its success to its worldwide work force and commitment to global diversity, which opens our proprietary software to innovative ideas from anywhere. Along the way, we have witnessed firsthand how employees, partners, and customers with diverse perspectives and experiences contribute to creative problem solving and better solutions for our customers and their businesses.



Senior Front End Engineer
US or Canada

OUR CUSTOMERS DEVELOP SOFTWARE AT THE SPEED OF IDEAS

CloudBees is powering the continuous economy by offering the world’s first end-to-end continuous software delivery management system (SDM). For millions of developers and product teams driving innovation for businesses large or small, SDM builds on continuous integration (CI) and continuous delivery (CD) to enable all functions and teams within and around the software delivery organization to best work together to amplify value creation.
 
CloudBees is the continuous integration (CI), continuous delivery (CD) and application release automation (ARA) powerhouse built from the commercial success of its products and its open source leadership as the largest contributor to Jenkins and a founding member of the Continuous Delivery Foundation (CDF). With a globally distributed workforce of more than 500 employees, the company reflects the global nature of the DevOps movement. We believe in walking the talk! From startups with full-stack developers practicing NoOps to large Fortune 100 companies, CloudBees enables all software-driven organizations to intelligently deploy the right capabilities at the right time.
  
Over 3,500 of the world’s best known brands and over 50% of the Fortune 500, invest in CloudBees because of its ability to work across any cloud, in any development environment and to balance corporate governance and control with developer flexibility and freedom.
 
CloudBees is home to the world’s leading DevOps experts helping thousands of companies harness the power of “continuous everything” and putting them on the fastest path from great idea, to great software, to great business value.

What You’ll Do

  • Join a small team developing CloudBees SaaS-based products
  • Architect new UI features or enhance existing features with a Ruby on Rails back end and a Vue front end
  • Collaborate with designers as well as engineers on other teams to continuously evolve and improve our internal component library
  • Work closely with product managers, designers, technical writers, and support engineers to ensure our CI/CD user experience is as simple and frictionless as possible
  • Mentor and influence team members through collaborative development
  • Review pull requests and provide feedback
  • Demonstrate flexibility by assisting with other projects when needed

Why CloudBees?

  • CloudBees is the hub of enterprise Jenkins and DevOps, providing the fastest path from ideas to software with smarter solutions for continuous development and delivery.
  • We build our business on Jenkins by offering enterprise solutions and professional services, and our customers range from the blue chips to small innovative companies.
  • We work on various projects in different areas. We are flexible about changing teams and assignments according to the interest and passion of engineers.
  • Our Engineering teams are distributed by design. You can work from one of our offices or remotely without feeling disconnected.
  • You have an opportunity to contribute to open-source projects and communities during your working time.
  • We work with a wide variety of emerging technologies, including Docker, Kubernetes, cloud computing platforms (Amazon Web Services, Pivotal, VMWare, etc.), ElasticSearch, etc.
  • We offer an attractive benefits package including stock options.

What The Role Requires

  • Strong background in front end application development (5+ years preferred)
  • Hands on experience with Vue or React (2+ years preferable)
  • General familiarity with continuous delivery and continuous deployment practices 
  • Strong desire to constantly learn new technologies with a general passion for improving developer productivity
  • Passionate about building great developer tooling user interfaces
  • Adaptability and a positive attitude
  • Bonus - experience integrating React and Vue components/Ruby on Rails experience 

What You’ll Get

  • Highly competitive benefits and vacation package.
  • Ability to work at a fast growing company with some of the most talented people in the industry.
  • Fun, hardworking, and casual environment.
  • Endless growth opportunities.
  • Team outings.

At CloudBees, we believe that the more diverse we are, the better we serve our customers.  A global community like Jenkins demands a global focus from CloudBees.  Organizations with greater diversity—gender, racial, ethnic, and global—are stronger partners to their customers.  Whether by creating more innovative products, or better understanding our worldwide customers, or establishing a stronger cross-section of cultural leadership skills, diversity strengthens all aspects of the CloudBees organization.

In the technology industry, diversity creates a competitive advantage.  CloudBees customers demand technologies from us that solve their software development, and therefore their business problems so that they can better serve their customers.  CloudBees attributes much of its success to its worldwide workforce and commitment to global diversity, which opens our proprietary software to innovative ideas from anywhere.  Along the way, we have witnessed firsthand how employees, partners, and customers with diverse perspectives and experiences contribute to creative problem solving and better solutions for our customers and their businesses.

You may also like