7 ms·

Principles for building and scaling feature flag systems

212 points by ferrantim 3y ago

sinuhe69 3y ago
I know features flags like the flags of FF, but they are in fact config files. How does one realize dynamic control of features flags? Are they using a kind of logic to determine to turn on/off a feature or do they query a central database to know that? Can someone explain its basic mechanism? Thanks
oddx 3y ago
I dedicated a day to evaluating feature flag software based on specific criteria: - Must support multiple SDKs, including Java and Ruby. - Should be self-hosted with PostgreSQL database support. - Needs to enable remote configuration for arbitrary values (not just feature flags). I don't run two separate services for this. - Should offer some UI functionality. - it should cache flag values locally and, ideally, provide live data updates (though pooling is acceptable). Here are the four options that met these basic criteria and underwent detailed evaluation: - Unleash: Impressive and powerful, but its UI is more complex than needed, and it lacks remote configuration. - Flagsmith: Offers remote configuration but appears less polished with some features not working smoothly; Java SDK error reporting needs improvement. - Flipt: Simple and elegant, but lacks remote configuration and local caching for Java SDK. - FeatureHub: Offers fewer features than Unleash and Flagsmith; its Java API seems somewhat enterprisly but supports remote configuration and live data updates. Currently, I'm leaning towards FeatureHub. If remote configuration isn't necessary, Unleash offers more features, and if simplicity is key and local caching isn't needed, Flipt is an attractive option.
- bullcitydev 3y ago
  Hey thanks for giving Flipt a look! I'm the creator of Flipt so would love to chat more about your needs to see how we could make it work for your use case! We're actively looking into providing local caching for all our SDKs btw and would love to learn more about what your requirements are for remote configuration as it's also on our radar! Feel free to send me an email at: mark (at) flipt.io.
dgorton 3y ago
Open Source knowledge sharing. I like it: "Unleash is open-source, and so are these principles. Have something to contribute? Open a PR or discussion on our Github."
jt2190 3y ago
I just item 1 (“Enable run-time control. Control flags dynamically, not using config files”) and it’s almost exclusively focused on what to do but not on why to do it. It seems to be skipping past the use-cases and assumptions, in particular, describing what a system with feature flags looks and acts like, what the benefits and drawbacks are.
- tmpX7dMeXU 3y ago
  Yeah, because it’s describing how a product works, being passed off as knowledge.
- ivarconr 3y ago
  > It seems to be skipping past the use-cases and assumptions This is a great feedback. Our intention was to describe how such a system work at scale, but I see we could do better in this section, thanks! Do you have some use-cases in mind?
tiberriver256 3y ago
Item #1 depends on the reason you're using feature flags. For a more nuanced and careful discussion of the topic I like to reference: https://martinfowler.com/articles/feature-toggles.html https://martinfowler.com/articles/feature-toggles.html
- gastonfournier 3y ago
  It's true that there are more long-lived use cases, but if you have the ability to choose, runtime controlled ones cover both cases, while compile time only cover some use cases. But fair point
zellyn 3y ago
Background: I work at Block/Square, on the team that owns (but didn't build) our internal Feature Flag system, and also have a lot of experience with using LaunchDarkly. I like the idea of caching locally, although k8s makes that a bit more difficult since containers are typically ephemeral. People will use feature flags for things that they shouldn't, so eventually "falling back go default values" will cause production problems. One thing you can do to help with this is run proxies closer to your services. For example, LaunchDarkly has an open source "Relay". Local evaluation seems to be pretty standard at this point, although I'd argue that delivering flag definitions is (relatively) easy. One of the real value-add of a product like LaunchDarkly is all the things they can do when your applications send evaluation data upstream: unused flags, only-ever-evaluated-to-the-default flags, only-ever-evaluated-to-one-outcome flags, etc. One best practice that I'd love to see spread (in our codebases too) is always naming the full feature flag directly in code, as a string (not a constant). I'd argue the same practice should be taken with metrics names. One of the most useful things to know (but seldom communicated clearly near landing pages) is a basic sketch of the architecture. It's necessary to know how things will behave if there is trouble. For instance: our internal system uses ZK to store (protobuf) flag definitions, and applications set watches to be notified of changes. LaunchDarkly clients download all flags[1] in the project on connection, then stream changes. If I were going to build a feature flag system, I would ensure that there is a global, incrementing counter that is updated every time any change is made, and make it a fundamental aspect of the design. That way, clients can cache what they've seen, and easily fetch only necessary updates. You could also imagine annotating that generation ID into W3C Baggage, and passing it through the microservices call graph to ensure evaluation at a consistent point in time (clients would need to cache history for a minute or two, of course). One other dimension in which feature flag services vary is by the complexity of the rules they allow you to evaluate. Our internal system has a mini expression language (probably overkill). LaunchDarkly's arguably better system gives you an ordered set of rules within which conditions are ANDed together. Both allow you to pass in arbitrary contexts of key/value pairs. Many open source solutions (Unleash, last I checked, some time ago) are more limited: some of them don't let you vary on inputs, some only a small set of prescribed attributes. I think the time is ripe for an open standard client API for feature flags. I think standardizing the communication mechanisms would be constricting, but there's no reason we couldn't create something analogous to (or even part of) the Open Telemetry client SDK for feature flags. If you are seriously interested in collaborating on that, please get in touch. (I'm "zellyn" just about everywhere) [1] Yes, this causes problems if you have too many flags in one project. They have a pretty nice filtering solution that's almost fully ready. [Update: edited to make 70% of it not italics ]
Lutger 3y ago
We use Unleash. There are many things you can do with feature flags and Unleash helps with a lot of them. However, my feeling is 80% of the value comes from 20% of the features. Even a much simpler system provides a ton of benefit. For me, it is top of the list after having automated tests and automated deployments.
mabbo 3y ago
> Make feature flags short-lived. Do not confuse flags with application configuration. This is my current battle. I introduced feature flags to the team as a means to separate deployment from launch of new features. For the sake of getting it working and used, I made the mis-step of backing the flags with config files with the intent to get Launch Darkly or Unleash working ASAP instead to replace them. Then another dev decided that these Feature Flags look like a great way to implement permanent application configs for different subsets of entities in our system. In fact, he evangelized it in his design for a major new project (I was not invited to the review). Now I have to stand back and watch as the feature flags are being used for long-term configurations. I objected when I saw the misuse- in a code review I said "hey that's not what these are for"- and was overruled by management. This is the design, there's no time to update it, I'm sure we can fix it later, someday. Lesson learned: make it very hard to misuse meta-features like feature flags, or someone will use them to get their stuff done faster.
- accountantbob 3y ago
  We did the same. We were early adopters of unleash and wrangled it to also host long term application configuration and even rule based application config. The architecture of unleash made it so simple to do in unleash vs having to evaluate, configure, and deploy a separate app config solution.
  - gastonfournier 3y ago
    Victim of your own success. As others were saying, when it works for short-lived its easy/no effort to use it for long-lived configurations.
- ivarconr 3y ago
  Thanks for sharing. I have seen systems grow in to thousands of flags, where most developers does not know what a particular flags do anymore.
- zellyn 3y ago
  Sadly, this is a battle you are destined to lose. I have almost completely given up. The best you can aim for is to use feature flags better rather than worse. - Some flags are going to stay forever: kill switches, load shedding, etc. (vendors are starting to incorporate this in the UI) - Unless you have a very-easy-to-use way to add arbitrary boolean feature toggles to individual user accounts (which can become its own mess), people are going to find it vastly easier to create feature flags with per-use override lists (almost all of them let you override on primary token). They will use your feature flags for: - Preview features: "is this user in the preview group?" - rollouts that might not ever go 100%: "should this organization use the old login flow?" - business-critical attributes that it would be a major incident to revert to defaults: "does this user operate under the alternate tax regime?" You can try to fight this (indeed, especially for that last one, you most definitely should!), but you will not ever completely win the feature flag ideological purity war!
aranchelk 3y ago
With regard to web-based services, once you’ve got the ability to do canary testing, IMO flags/toggles are less compelling — busier code and logic you’ll have to pull out later.
- erik_seaberg 3y ago
  Canarying gets you a 1/n treatment group, but it might be skew geographically (all affected users are near the canary’s datacenter). You need a percentage in a feature flag if 1/n is too big and you want, e.g., 0.1% of traffic.
- gastonfournier 3y ago
  I agree that if you have only a few changes going to prod, fast and doing canary testing, you should be covered. In my experience that's rarely the case because of multiple teams deploying changes at the same time, and even deployments in external services causing side effects in other services.
  - aranchelk 3y ago
    Emergent inter-service issues are challenging to deal with regardless. I’ve absolutely seen canary testing work in large environments with a lot of teams doing frequent deploys. The teams need to have the tooling to conduct their own canary testing and monitoring. As soon as you’re involving external services or anything persistent you may not be able to undo the damage of misbehaving software by simply disabling the offending code with a flag. In practice the cost/benefit of feature flags has never proven out for me, better to just speed up your deploys/rollbacks, the caveat is I’ve only ever worked in web environments, I can imagine with software running on an end user device it could solve some difficult problems provided you have a way to toggle the flag.
- baq 3y ago
  OTOH a flag gives you an ability to deploy and revert independent of the product’s release cycle.
staplung 3y ago
This feels a bit like the dicta on 12 Factor: rules handed down from a presumed authority without any discussion of the tradeoffs. Engineering is tradeoff evaluation. Give me some discussion about the alternatives, when and why they're inferior and don't pretend like the proposed solution doesn't have shortcomings or pitfalls.
- imiric 3y ago
  I agree with you that tradeoff evaluation is crucial in engineering, but I don't see the 12 Factor methodology as a set of strict rules. They're more like guidelines that are generally a good idea to follow for building modern applications or services. Some of the suggestions apply for any type of software, like having a single version controlled codebase, separate build/release/run stages, and using stateless processes. So it's good to be aware of _why_ those guidelines are considered a good thing, but as with any methodology, an engineer should be pragmatic in deciding when to follow it strictly, and when to adapt or ignore some of it. That said, I wouldn't want to work on software that completely ignores 12 Factor.
angarg12 3y ago
Offtopic but relevant: TL;DR if you break long posts into pages, at least have an option to see the whole thing in a single page. I use a browser extension to send websites to my Kindle. It's great for long-ish format blog posts that I want to read, but I don't have the time at the moment. However, whenever I see long blog posts that are broken into sections, each one in it's own page, it becomes a mess. It forces me to navigate each individual page and send it to my Kindle. Then in the Kindle I have a long list of unsorted files that I need to jump around to read in order. I understand breaking long pieces of text into pages makes it neater and more organized, but at least have an option to see the whole thing in a single page, as a way to export it somewhere else for easy reading.
rubicon33 3y ago
As an engineer, I am generally against feature flags. They fracture your code base, are sometimes never removed, and add complexity and logic that at best is a boolean check and at worse is something more involved. I'd love a world where engineers are given time to complete their feature in its entirety, and the feature is released when it is ready. Sadly, we do not live in that world and hence: feature flags.
- hn_throwaway_99 3y ago
  This misses the point. A big point of feature flags is that you don't yet know how features will be perceived until you get them in front of real users. I get what you'd like "as an engineer", but it ignores the needs of the business.
  - rubicon33 3y ago
    Isn't that the job a product manager? There are other means and methodologies for gathering user sentiment before you go and build something. You should get as close as you can, release the product, and iterate. Todays world is release the product in some ramshackle form or fashion, collect feedback, iterate. To do that introduces a new construct of Feature Flags that would otherwise not be necessary.
    - baq 3y ago
      Yeah product manager says ‘run this on 5% of traffic for 2 weeks and then we’ll see what the next iteration should be’.
      - Rapzid 3y ago
        "Also, if one customer is having a particularly bad time we need to be able to disable the feature for them while continuing to collect feedback from everyone else."
        mrcarrot 3y ago
jacomoRodriguez 3y ago
Is it just me or does this articles text structure and wording strongly indicate that is was written by gpt?
- gingerrr 3y ago
  Definitely getting strong uncanny valley prose vibes. Hard to tell if it's generated or written in an attempt to be as plain English as possible, but either way feels strangely vacuous for a technical opinion piece. There's no writer's voice.
  - flitzofolov 3y ago
    It's not really an opinion piece is it? It's docs. The language seems appropriate for articulating principles.
    - gingerrr 3y ago
      Whoops I missed this reply! I think it's absolutely an opinion piece - defining specific items as principles by definition means expressing opinionated ideas about the relative priority of those items over others. Also, imperative mood contains value judgment, which is inherently opinion-based (e.g. "Never expose PII"). Making arguments for why you should or should not do things requires expressing opinions about relative importance, weight etc. If this were instead an article describing what feature flags are, or one performing a survey of various approaches to building/scaling them, I think the lack of voice is just fine - that's dealing in statement of fact. But this article mandates and implores and exhorts - the value judgments inherent in that pathos are empty without genuine authorship. Also I'm not saying the lack of voice is bad even for conveying meaning or teaching - more that it is jarring and uncanny to read imperative claims in an empty robotic voice devoid of ethos. Finally, I also might be biased by my first documentation love, the zeromq guide, which is an extremely-strongly-opinionated piece of docs that does its job exceptionally well. I think when writing about how or why, a strong writer's voice is more compelling. This article stretches past just the what into those other question words, so its seeming lack of authorial authority falls flat to me. Thanks for giving me an excuse to blabble lol.
adasdasdas 3y ago
More principles - Require in code defaults for fault tolerance - Start annoying the flag author to delete if the flag is over a month old - Partial rollout should be by hash on user id - Contextual flag features should always be supplied by client (e.g. only show in LA, the location should be provided by client)
- hamandcheese 3y ago
  > Partial rollout should be by hash on user id With a per-flag salt as well, otherwise the same user will always have bad luck and be subject to experiments first.
  - zellyn 3y ago
    Our in-house solution hashes flagname+key, and LD does the same but adds salt
- tantalor 3y ago
  > Start annoying the flag author to delete if the flag is over a month old No problem, filter that email directly to spam folder.
- stravant 3y ago
  Better yet require always the same default for boolean flags, so that it's easier to reason about lifecycle for them.
dabeeeenster 3y ago
For those that dont know about the project, check out Open Feature https://openfeature.dev/ https://openfeature.dev/ which is sort of like Open Telemetry but for feature flags. Helps avoid vendor lock in. We're a young project and looking for help and to build the community!
- triyambakam 3y ago
  Wow, part of the CNCF. That's awesome
eximius 3y ago
The system we're building now meets most of these but not necessarily in the way described. First, we're building a runtime configuration system on top of AWS AppConfig. YAML/proto validation that pushes to AppConfig via gitops and bazel. Configurations are namespaced so the unique names is solved. It's all open in git. Feature flags are special cases of runtime configuration. We are distinguishing backend feature flags from experimentation/variants for users. We don't have (or want) cohorting by user IDs or roles. We have a separate system for that and it does it well. The last two points - distinguishing between experimentation/feature variants and feature flags as runtime configuration are somewhat axiomatic differences. Folks might disagree but ultimately we have that separate system that solves that case. They're complimentary and share a lot of properties but ultimately it solves a lot of angst if you don't force both to be the same tool.
tvink 3y ago
>Organizations who adopt feature flags see improvements in all key operational metrics for DevOps: Lead time to changes, mean-time-to-recovery, deployment frequency, and change failure rate. Is this true? unfortunately there's no sources indicated, and a quick check on scholar doesn't show me anything of the sort.
- gastonfournier 3y ago
  There are a few case studies listed in most of the feature flag solutions, of course, each organization is completely different and the maturity of each organization varies. But feature flags are a 2-way-door decision, meaning that you can adopt them at smaller scale, try it out and see if it works for you before making a decision. Here's a list of case studies from some of the solutions referred in the comments, some focus on operational metrics, others in lead time to changes: https://www.getunleash.io/case-studies https://www.getunleash.io/case-studies https://launchdarkly.com/case-studies/ https://launchdarkly.com/case-studies/ https://www.flagsmith.com/case-studies https://www.flagsmith.com/case-studies
- timothyfcook 3y ago
  The LaunchDarkly 2022 State of Feature Management Report has results from surveying 1000 software people and looks at impact on those DevOps metrics: https://launchdarkly.com/state-of-feature-management/ https://launchdarkly.com/state-of-feature-management/
- getrealyall 3y ago
  This was written by a company selling feature flag software. Use that information as thou wilt.
flitzofolov 3y ago
I couldn't find an easy link from these docs to the product page on mobile. Seems like a wasted opportunity. I had to edit the URL to get to the company website.

212 points by ferrantim 3y ago

I know features flags like the flags of FF, but they are in fact config files. How does one realize dynamic control of features flags? Are they using a kind of logic to determine to turn on/off a feature or do they query a central database to know that? Can someone explain its basic mechanism? Thanks

oddx 3y ago

I dedicated a day to evaluating feature flag software based on specific criteria: - Must support multiple SDKs, including Java and Ruby. - Should be self-hosted with PostgreSQL database support. - Needs to enable remote configuration for arbitrary values (not just feature flags). I don't run two separate services for this. - Should offer some UI functionality. - it should cache flag values locally and, ideally, provide live data updates (though pooling is acceptable). Here are the four options that met these basic criteria and underwent detailed evaluation: - Unleash: Impressive and powerful, but its UI is more complex than needed, and it lacks remote configuration. - Flagsmith: Offers remote configuration but appears less polished with some features not working smoothly; Java SDK error reporting needs improvement. - Flipt: Simple and elegant, but lacks remote configuration and local caching for Java SDK. - FeatureHub: Offers fewer features than Unleash and Flagsmith; its Java API seems somewhat enterprisly but supports remote configuration and live data updates. Currently, I'm leaning towards FeatureHub. If remote configuration isn't necessary, Unleash offers more features, and if simplicity is key and local caching isn't needed, Flipt is an attractive option.

bullcitydev 3y ago
Hey thanks for giving Flipt a look! I'm the creator of Flipt so would love to chat more about your needs to see how we could make it work for your use case! We're actively looking into providing local caching for all our SDKs btw and would love to learn more about what your requirements are for remote configuration as it's also on our radar! Feel free to send me an email at: mark (at) flipt.io.

dgorton 3y ago

Open Source knowledge sharing. I like it: "Unleash is open-source, and so are these principles. Have something to contribute? Open a PR or discussion on our Github."

jt2190 3y ago

I just item 1 (“Enable run-time control. Control flags dynamically, not using config files”) and it’s almost exclusively focused on what to do but not on why to do it. It seems to be skipping past the use-cases and assumptions, in particular, describing what a system with feature flags looks and acts like, what the benefits and drawbacks are.

tmpX7dMeXU 3y ago
Yeah, because it’s describing how a product works, being passed off as knowledge.
ivarconr 3y ago
> It seems to be skipping past the use-cases and assumptions This is a great feedback. Our intention was to describe how such a system work at scale, but I see we could do better in this section, thanks! Do you have some use-cases in mind?

tiberriver256 3y ago

Item #1 depends on the reason you're using feature flags. For a more nuanced and careful discussion of the topic I like to reference: https://martinfowler.com/articles/feature-toggles.html https://martinfowler.com/articles/feature-toggles.html

gastonfournier 3y ago
It's true that there are more long-lived use cases, but if you have the ability to choose, runtime controlled ones cover both cases, while compile time only cover some use cases. But fair point

zellyn 3y ago

Background: I work at Block/Square, on the team that owns (but didn't build) our internal Feature Flag system, and also have a lot of experience with using LaunchDarkly. I like the idea of caching locally, although k8s makes that a bit more difficult since containers are typically ephemeral. People will use feature flags for things that they shouldn't, so eventually "falling back go default values" will cause production problems. One thing you can do to help with this is run proxies closer to your services. For example, LaunchDarkly has an open source "Relay". Local evaluation seems to be pretty standard at this point, although I'd argue that delivering flag definitions is (relatively) easy. One of the real value-add of a product like LaunchDarkly is all the things they can do when your applications send evaluation data upstream: unused flags, only-ever-evaluated-to-the-default flags, only-ever-evaluated-to-one-outcome flags, etc. One best practice that I'd love to see spread (in our codebases too) is always naming the full feature flag directly in code, as a string (not a constant). I'd argue the same practice should be taken with metrics names. One of the most useful things to know (but seldom communicated clearly near landing pages) is a basic sketch of the architecture. It's necessary to know how things will behave if there is trouble. For instance: our internal system uses ZK to store (protobuf) flag definitions, and applications set watches to be notified of changes. LaunchDarkly clients download all flags[1] in the project on connection, then stream changes. If I were going to build a feature flag system, I would ensure that there is a global, incrementing counter that is updated every time any change is made, and make it a fundamental aspect of the design. That way, clients can cache what they've seen, and easily fetch only necessary updates. You could also imagine annotating that generation ID into W3C Baggage, and passing it through the microservices call graph to ensure evaluation at a consistent point in time (clients would need to cache history for a minute or two, of course). One other dimension in which feature flag services vary is by the complexity of the rules they allow you to evaluate. Our internal system has a mini expression language (probably overkill). LaunchDarkly's arguably better system gives you an ordered set of rules within which conditions are ANDed together. Both allow you to pass in arbitrary contexts of key/value pairs. Many open source solutions (Unleash, last I checked, some time ago) are more limited: some of them don't let you vary on inputs, some only a small set of prescribed attributes. I think the time is ripe for an open standard client API for feature flags. I think standardizing the communication mechanisms would be constricting, but there's no reason we couldn't create something analogous to (or even part of) the Open Telemetry client SDK for feature flags. If you are seriously interested in collaborating on that, please get in touch. (I'm "zellyn" just about everywhere) [1] Yes, this causes problems if you have too many flags in one project. They have a pretty nice filtering solution that's almost fully ready. [Update: edited to make 70% of it not italics ]

Lutger 3y ago

We use Unleash. There are many things you can do with feature flags and Unleash helps with a lot of them. However, my feeling is 80% of the value comes from 20% of the features. Even a much simpler system provides a ton of benefit. For me, it is top of the list after having automated tests and automated deployments.

mabbo 3y ago

> Make feature flags short-lived. Do not confuse flags with application configuration. This is my current battle. I introduced feature flags to the team as a means to separate deployment from launch of new features. For the sake of getting it working and used, I made the mis-step of backing the flags with config files with the intent to get Launch Darkly or Unleash working ASAP instead to replace them. Then another dev decided that these Feature Flags look like a great way to implement permanent application configs for different subsets of entities in our system. In fact, he evangelized it in his design for a major new project (I was not invited to the review). Now I have to stand back and watch as the feature flags are being used for long-term configurations. I objected when I saw the misuse- in a code review I said "hey that's not what these are for"- and was overruled by management. This is the design, there's no time to update it, I'm sure we can fix it later, someday. Lesson learned: make it very hard to misuse meta-features like feature flags, or someone will use them to get their stuff done faster.

accountantbob 3y ago
We did the same. We were early adopters of unleash and wrangled it to also host long term application configuration and even rule based application config. The architecture of unleash made it so simple to do in unleash vs having to evaluate, configure, and deploy a separate app config solution.
- gastonfournier 3y ago
  Victim of your own success. As others were saying, when it works for short-lived its easy/no effort to use it for long-lived configurations.
ivarconr 3y ago
Thanks for sharing. I have seen systems grow in to thousands of flags, where most developers does not know what a particular flags do anymore.
zellyn 3y ago
Sadly, this is a battle you are destined to lose. I have almost completely given up. The best you can aim for is to use feature flags better rather than worse. - Some flags are going to stay forever: kill switches, load shedding, etc. (vendors are starting to incorporate this in the UI) - Unless you have a very-easy-to-use way to add arbitrary boolean feature toggles to individual user accounts (which can become its own mess), people are going to find it vastly easier to create feature flags with per-use override lists (almost all of them let you override on primary token). They will use your feature flags for: - Preview features: "is this user in the preview group?" - rollouts that might not ever go 100%: "should this organization use the old login flow?" - business-critical attributes that it would be a major incident to revert to defaults: "does this user operate under the alternate tax regime?" You can try to fight this (indeed, especially for that last one, you most definitely should!), but you will not ever completely win the feature flag ideological purity war!

aranchelk 3y ago

With regard to web-based services, once you’ve got the ability to do canary testing, IMO flags/toggles are less compelling — busier code and logic you’ll have to pull out later.

erik_seaberg 3y ago
Canarying gets you a 1/n treatment group, but it might be skew geographically (all affected users are near the canary’s datacenter). You need a percentage in a feature flag if 1/n is too big and you want, e.g., 0.1% of traffic.
gastonfournier 3y ago
I agree that if you have only a few changes going to prod, fast and doing canary testing, you should be covered. In my experience that's rarely the case because of multiple teams deploying changes at the same time, and even deployments in external services causing side effects in other services.
- aranchelk 3y ago
  Emergent inter-service issues are challenging to deal with regardless. I’ve absolutely seen canary testing work in large environments with a lot of teams doing frequent deploys. The teams need to have the tooling to conduct their own canary testing and monitoring. As soon as you’re involving external services or anything persistent you may not be able to undo the damage of misbehaving software by simply disabling the offending code with a flag. In practice the cost/benefit of feature flags has never proven out for me, better to just speed up your deploys/rollbacks, the caveat is I’ve only ever worked in web environments, I can imagine with software running on an end user device it could solve some difficult problems provided you have a way to toggle the flag.
baq 3y ago
OTOH a flag gives you an ability to deploy and revert independent of the product’s release cycle.

staplung 3y ago

This feels a bit like the dicta on 12 Factor: rules handed down from a presumed authority without any discussion of the tradeoffs. Engineering is tradeoff evaluation. Give me some discussion about the alternatives, when and why they're inferior and don't pretend like the proposed solution doesn't have shortcomings or pitfalls.

imiric 3y ago
I agree with you that tradeoff evaluation is crucial in engineering, but I don't see the 12 Factor methodology as a set of strict rules. They're more like guidelines that are generally a good idea to follow for building modern applications or services. Some of the suggestions apply for any type of software, like having a single version controlled codebase, separate build/release/run stages, and using stateless processes. So it's good to be aware of _why_ those guidelines are considered a good thing, but as with any methodology, an engineer should be pragmatic in deciding when to follow it strictly, and when to adapt or ignore some of it. That said, I wouldn't want to work on software that completely ignores 12 Factor.

angarg12 3y ago

Offtopic but relevant: TL;DR if you break long posts into pages, at least have an option to see the whole thing in a single page. I use a browser extension to send websites to my Kindle. It's great for long-ish format blog posts that I want to read, but I don't have the time at the moment. However, whenever I see long blog posts that are broken into sections, each one in it's own page, it becomes a mess. It forces me to navigate each individual page and send it to my Kindle. Then in the Kindle I have a long list of unsorted files that I need to jump around to read in order. I understand breaking long pieces of text into pages makes it neater and more organized, but at least have an option to see the whole thing in a single page, as a way to export it somewhere else for easy reading.

rubicon33 3y ago

As an engineer, I am generally against feature flags. They fracture your code base, are sometimes never removed, and add complexity and logic that at best is a boolean check and at worse is something more involved. I'd love a world where engineers are given time to complete their feature in its entirety, and the feature is released when it is ready. Sadly, we do not live in that world and hence: feature flags.

hn_throwaway_99 3y ago
This misses the point. A big point of feature flags is that you don't yet know how features will be perceived until you get them in front of real users. I get what you'd like "as an engineer", but it ignores the needs of the business.
- rubicon33 3y ago
  Isn't that the job a product manager? There are other means and methodologies for gathering user sentiment before you go and build something. You should get as close as you can, release the product, and iterate. Todays world is release the product in some ramshackle form or fashion, collect feedback, iterate. To do that introduces a new construct of Feature Flags that would otherwise not be necessary.
  - baq 3y ago
    Yeah product manager says ‘run this on 5% of traffic for 2 weeks and then we’ll see what the next iteration should be’.
    - Rapzid 3y ago
      "Also, if one customer is having a particularly bad time we need to be able to disable the feature for them while continuing to collect feedback from everyone else."
      - mrcarrot 3y ago

jacomoRodriguez 3y ago

Is it just me or does this articles text structure and wording strongly indicate that is was written by gpt?

gingerrr 3y ago
Definitely getting strong uncanny valley prose vibes. Hard to tell if it's generated or written in an attempt to be as plain English as possible, but either way feels strangely vacuous for a technical opinion piece. There's no writer's voice.
- flitzofolov 3y ago
  It's not really an opinion piece is it? It's docs. The language seems appropriate for articulating principles.
  - gingerrr 3y ago
    Whoops I missed this reply! I think it's absolutely an opinion piece - defining specific items as principles by definition means expressing opinionated ideas about the relative priority of those items over others. Also, imperative mood contains value judgment, which is inherently opinion-based (e.g. "Never expose PII"). Making arguments for why you should or should not do things requires expressing opinions about relative importance, weight etc. If this were instead an article describing what feature flags are, or one performing a survey of various approaches to building/scaling them, I think the lack of voice is just fine - that's dealing in statement of fact. But this article mandates and implores and exhorts - the value judgments inherent in that pathos are empty without genuine authorship. Also I'm not saying the lack of voice is bad even for conveying meaning or teaching - more that it is jarring and uncanny to read imperative claims in an empty robotic voice devoid of ethos. Finally, I also might be biased by my first documentation love, the zeromq guide, which is an extremely-strongly-opinionated piece of docs that does its job exceptionally well. I think when writing about how or why, a strong writer's voice is more compelling. This article stretches past just the what into those other question words, so its seeming lack of authorial authority falls flat to me. Thanks for giving me an excuse to blabble lol.

adasdasdas 3y ago

More principles - Require in code defaults for fault tolerance - Start annoying the flag author to delete if the flag is over a month old - Partial rollout should be by hash on user id - Contextual flag features should always be supplied by client (e.g. only show in LA, the location should be provided by client)

hamandcheese 3y ago
> Partial rollout should be by hash on user id With a per-flag salt as well, otherwise the same user will always have bad luck and be subject to experiments first.
- zellyn 3y ago
  Our in-house solution hashes flagname+key, and LD does the same but adds salt
tantalor 3y ago
> Start annoying the flag author to delete if the flag is over a month old No problem, filter that email directly to spam folder.
stravant 3y ago
Better yet require always the same default for boolean flags, so that it's easier to reason about lifecycle for them.

dabeeeenster 3y ago

For those that dont know about the project, check out Open Feature https://openfeature.dev/ https://openfeature.dev/ which is sort of like Open Telemetry but for feature flags. Helps avoid vendor lock in. We're a young project and looking for help and to build the community!

triyambakam 3y ago
Wow, part of the CNCF. That's awesome

eximius 3y ago

The system we're building now meets most of these but not necessarily in the way described. First, we're building a runtime configuration system on top of AWS AppConfig. YAML/proto validation that pushes to AppConfig via gitops and bazel. Configurations are namespaced so the unique names is solved. It's all open in git. Feature flags are special cases of runtime configuration. We are distinguishing backend feature flags from experimentation/variants for users. We don't have (or want) cohorting by user IDs or roles. We have a separate system for that and it does it well. The last two points - distinguishing between experimentation/feature variants and feature flags as runtime configuration are somewhat axiomatic differences. Folks might disagree but ultimately we have that separate system that solves that case. They're complimentary and share a lot of properties but ultimately it solves a lot of angst if you don't force both to be the same tool.

tvink 3y ago

>Organizations who adopt feature flags see improvements in all key operational metrics for DevOps: Lead time to changes, mean-time-to-recovery, deployment frequency, and change failure rate. Is this true? unfortunately there's no sources indicated, and a quick check on scholar doesn't show me anything of the sort.

gastonfournier 3y ago
There are a few case studies listed in most of the feature flag solutions, of course, each organization is completely different and the maturity of each organization varies. But feature flags are a 2-way-door decision, meaning that you can adopt them at smaller scale, try it out and see if it works for you before making a decision. Here's a list of case studies from some of the solutions referred in the comments, some focus on operational metrics, others in lead time to changes: https://www.getunleash.io/case-studies https://www.getunleash.io/case-studies https://launchdarkly.com/case-studies/ https://launchdarkly.com/case-studies/ https://www.flagsmith.com/case-studies https://www.flagsmith.com/case-studies
timothyfcook 3y ago
The LaunchDarkly 2022 State of Feature Management Report has results from surveying 1000 software people and looks at impact on those DevOps metrics: https://launchdarkly.com/state-of-feature-management/ https://launchdarkly.com/state-of-feature-management/
getrealyall 3y ago
This was written by a company selling feature flag software. Use that information as thou wilt.

flitzofolov 3y ago

I couldn't find an easy link from these docs to the product page on mobile. Seems like a wasted opportunity. I had to edit the URL to get to the company website.