5 ms·

I 100% agree with everything you said and more. I spent a significant amount of time at my job arguing constantly with the entire engineering department that “F

by scott_w 14d ago

I 100% agree with everything you said and more. I spent a significant amount of time at my job arguing constantly with the entire engineering department that “Feature Flags” are not all the same thing and had to push back on efforts to mix them all up. Examples: Experiment flags for A/B testing Permission checks at a user level Product flags for feature gating by plan Rollout flags to launch a new feature gradually Configuration for an account/system/feature The number of times I had to push against the “just put it behind hasFeatureFlag(user, flag)” is more than I can count at this point! I think it comes from a misunderstanding of the DRY principle, to be honest.

esafak 14d ago
Why would you not use the same service to handle them? Less is more.
- hasyimibhar 14d ago
  Having worked at a large company that misuses feature flag service for everything, here are some examples why you shouldn’t: - one team uses feature flag for product gating. Feature flag service goes down. Users temporarily got locked out of the features they paid for. - one team uses feature flag for dynamic pricing by leveraging targeting rules (how hard is it to write a bunch of if else in code?). It’s evaluated against all users, even if they are not active (for analysis reasons). Feature flag service charges by MAU. We have millions of users. Our feature flag service bill is now 6 digits per year. - one team uses feature flag as literal json store instead of a proper db (god knows why). Someone updated the value but the “schema” is wrong. Shit breaks.
  - esafak 14d ago
    > - one team uses feature flag for product gating. Feature flag service goes down. Users temporarily got locked out of the features they paid for. That could happen if they used a separate service. In this case you need to build defaults; open or closed, as the case demands. > - one team uses feature flag for dynamic pricing by leveraging targeting rules (how hard is it to write a bunch of if else in code?). It’s evaluated against all users, even if they are not active (for analysis reasons). Feature flag service charges by MAU. We have millions of users. Our feature flag service bill is now 6 digits per year. This is a valid reason if you don't own the service, as it seems you don't. If you did, you should have asked your internal customers what their requirements were. > - one team uses feature flag as literal json store instead of a proper db (god knows why). Someone updated the value but the “schema” is wrong. Shit breaks. Separating the services would not help here. They are simply using the wrong tool.
    - jameshart 14d ago
      Golden rule of large scale software dev: any system you expose that allows other teams to write text to it will eventually be used by another team as an ad hoc JSON database
  - ceased 13d ago
    assuming you're on launchdarkly just switch to unleash, they charge us per seat lot cheaper
- scott_w 14d ago
  I explained: they’re all doing different things, they have different behaviours, they have different performance and uptime characteristics. The reason that the values change are completely different, so how that information is set and read will be completely different. Who sets it, who reads it. An example: at my current job, we’re using LaunchDarkly for config management, not just rollout. This means an incident on LaunchDarkly didn’t just stop a rollout, it caused the system behaviour to change for a large number of customers as it reverted to the default.
jdwyah 14d ago
oo yeah. entitlements is a huge one. Sooooo many orgs are in an absolute rats nest of confusion because half their entitlements are in something like an auth system, half are in something like billing and the third half is hacked into feature flags. One particular pain point is that finance or something "do users with access to feature X retain better and it's super hard to figure out. My overall feel is that developers are trying to solve a problem and that it turns out the configuration is as fundamentally as important as the software itself. I define "dynamic configuration" as "a key value store which takes context and has rules based system to give you a value". This primitive turns out to be extraordinarily useful and powerful. Rather than try to split it into N different systems, each custom / don't have telemetry / aren't available to all services SDKs. What if you have ONE BIG CONFIG system that really does a bang up job. (Note: copying the big boys is not always the right move, but https://research.facebook.com/publications/holistic-configuration-management-at-facebook/ https://research.facebook.com/publications/holistic-configur... does give some credence that this is a decent approach) By focussing on one terrific system, you can put all your eggs into the basket of making that reliable / comprehensible / flexible / strict. The one on your list that I think DOESN'T belong would be permission checks at user level. The N of that is not a good fit for config (though when you look at the facebook paper, it's pretty wild how they've scaled it (but it doesn't do permissions afaik)).

by scott_w 14d ago

esafak 14d ago

Why would you not use the same service to handle them? Less is more.

hasyimibhar 14d ago
Having worked at a large company that misuses feature flag service for everything, here are some examples why you shouldn’t: - one team uses feature flag for product gating. Feature flag service goes down. Users temporarily got locked out of the features they paid for. - one team uses feature flag for dynamic pricing by leveraging targeting rules (how hard is it to write a bunch of if else in code?). It’s evaluated against all users, even if they are not active (for analysis reasons). Feature flag service charges by MAU. We have millions of users. Our feature flag service bill is now 6 digits per year. - one team uses feature flag as literal json store instead of a proper db (god knows why). Someone updated the value but the “schema” is wrong. Shit breaks.
- esafak 14d ago
  > - one team uses feature flag for product gating. Feature flag service goes down. Users temporarily got locked out of the features they paid for. That could happen if they used a separate service. In this case you need to build defaults; open or closed, as the case demands. > - one team uses feature flag for dynamic pricing by leveraging targeting rules (how hard is it to write a bunch of if else in code?). It’s evaluated against all users, even if they are not active (for analysis reasons). Feature flag service charges by MAU. We have millions of users. Our feature flag service bill is now 6 digits per year. This is a valid reason if you don't own the service, as it seems you don't. If you did, you should have asked your internal customers what their requirements were. > - one team uses feature flag as literal json store instead of a proper db (god knows why). Someone updated the value but the “schema” is wrong. Shit breaks. Separating the services would not help here. They are simply using the wrong tool.
  - jameshart 14d ago
    Golden rule of large scale software dev: any system you expose that allows other teams to write text to it will eventually be used by another team as an ad hoc JSON database
- ceased 13d ago
  assuming you're on launchdarkly just switch to unleash, they charge us per seat lot cheaper
scott_w 14d ago
I explained: they’re all doing different things, they have different behaviours, they have different performance and uptime characteristics. The reason that the values change are completely different, so how that information is set and read will be completely different. Who sets it, who reads it. An example: at my current job, we’re using LaunchDarkly for config management, not just rollout. This means an incident on LaunchDarkly didn’t just stop a rollout, it caused the system behaviour to change for a large number of customers as it reverted to the default.

jdwyah 14d ago

oo yeah. entitlements is a huge one. Sooooo many orgs are in an absolute rats nest of confusion because half their entitlements are in something like an auth system, half are in something like billing and the third half is hacked into feature flags. One particular pain point is that finance or something "do users with access to feature X retain better and it's super hard to figure out. My overall feel is that developers are trying to solve a problem and that it turns out the configuration is as fundamentally as important as the software itself. I define "dynamic configuration" as "a key value store which takes context and has rules based system to give you a value". This primitive turns out to be extraordinarily useful and powerful. Rather than try to split it into N different systems, each custom / don't have telemetry / aren't available to all services SDKs. What if you have ONE BIG CONFIG system that really does a bang up job. (Note: copying the big boys is not always the right move, but https://research.facebook.com/publications/holistic-configuration-management-at-facebook/ https://research.facebook.com/publications/holistic-configur... does give some credence that this is a decent approach) By focussing on one terrific system, you can put all your eggs into the basket of making that reliable / comprehensible / flexible / strict. The one on your list that I think DOESN'T belong would be permission checks at user level. The N of that is not a good fit for config (though when you look at the facebook paper, it's pretty wild how they've scaled it (but it doesn't do permissions afaik)).